1{2 "document_id": "lb_qasper_5f00d4f6e62f4b99484eb78491f803f8143cc1b13ad33816",3 "family": "public_qasper",4 "split": "test",5 "text": "Introduction\nLanguage modelling in its inception had one-hot vector encoding of words. However, it captures only alphabetic ordering but not the word semantic similarity. Vector space models helps to learn word representations in a lower dimensional space and also captures semantic similarity. Learning word embedding aids in natural language processing tasks such as question answering and reasoning BIBREF0, stance detection BIBREF1, claim verification BIBREF2.\nRecent models BIBREF3, BIBREF4 work on the basis that words with similar context share semantic similarity. BIBREF4 proposes a neural probabilistic model which models the target word probability conditioned on the previous words using a recurrent neural network. Word2Vec models BIBREF3 such as continuous bag-of-words (CBOW) predict the target word given the context, and skip-gram model works in reverse of predicting the context given the target word. While, GloVe embeddings were based on a Global matrix factorization on local contexts BIBREF5. However, the aforementioned models do not handle words with multiple meanings (polysemies).\nBIBREF6 proposes a neural network approach considering both local and global contexts in learning word embeddings (point estimates). Their multiple prototype model handles polysemous words by providing apriori heuristics about word senses in the dataset. BIBREF7 proposes an alternative to handle polysemous words by a modified skip-gram model and EM algorithm. BIBREF8 presents a non-parametric based alternative to handle polysemies. However, these approaches fail to consider entailment relations among the words. BIBREF9 learn a Gaussian distribution per word using the expected likelihood kernel. However, for polysemous words, this may lead to word distributions with larger variances as it may have to cover various senses.\nBIBREF10 proposes multimodal word distribution approach. It captures polysemy. However, the energy based objective function fails to consider asymmetry and hence entailment. Textual entailment recognition is necessary to capture lexical inference relations such as causality (for example, mosquito $\\rightarrow $ malaria), hypernymy (for example, dog $\\models $ animal) etc.\nIn this paper, we propose to obtain multi-sense word embedding distributions by using a variant of max margin objective based on the asymmetric KL divergence energy function to capture textual entailment. Multi-sense distributions are advantageous in capturing polysemous nature of words and in reducing the uncertainty per word by distributing it across senses. However, computing KL divergence between mixtures of Gaussians is intractable, and we use a KL divergence approximation based on stricter upper and lower bounds. While capturing textual entailment (asymmetry), we have also not compromised on capturing symmetrical similarity between words (for example, funny and hilarious) which will be elucidated in Section $3.1$. We also show the effectiveness of the proposed approach on the benchmark word similarity and entailment datasets in the experimental section.\nMethodology ::: Word Representation\nProbabilistic representation of words helps one model uncertainty in word representation, and polysemy. Given a corpus $V$, containing a list of words each represented as $w$, the probability density for a word $w$ can be represented as a mixture of Gaussians with $C$ components BIBREF10.\nHere, $p_{w,j}$ represents the probability of word $w$ belonging to the component $j$, $\\operatorname{\\mathbf {\\mu }}_{w,j}$ represents $D$ dimensional word representation corresponding to the $j^{th}$ component sense of the word $w$, and $\\Sigma _{w,j}$ represents the uncertainty in representation for word $w$ belonging to component $j$.\nObjective function\nThe model parameters (means, covariances and mixture weights) $\\theta $ can be learnt using a variant of max-margin objective BIBREF11.\nHere $E_\\theta (\\cdot , \\cdot )$ represents an energy function which assigns a score to the pair of words, $w$ is a particular word under consideration, $c$ its positive context (same context), and $c^{\\prime }$ the negative context. The objective aims to push the margin of the difference between the energy function of a word $w$ to its positive context $c$ higher than its negative context $c$ by a threshold of $m$. Thus, word pairs in the same context gets a higher energy than the word pairs in the dissimilar context. BIBREF10 consider the energy function to be an expected likelihood kernel which is defined as follows.\nThis is similar to the cosine similarity metric over vectors and the energy between two words is maximum when they have similar distributions. But, the expected likelihood kernel is a symmetric metric which will not be suitable for capturing ordering among words and hence entailment.\nObjective function ::: Proposed Energy function\nAs each word is represented by a mixture of Gaussian distributions, KL divergence is a better choice of energy function to capture distance between distributions. Since, KL divergence is minimum when the distributions are similar and maximum when they are dissimilar, energy function is taken as exponentiated negative KL divergence.\nHowever, computing KL divergence between Gaussian mixtures is intractable and obtaining exact KL value is not possible. One way of approximating the KL is by Monte-Carlo approximation but it requires large number of samples to get a good approximation and is computationally expensive on high dimensional embedding space.\nAlternatively, BIBREF12 presents a KL approximation between Gaussian mixtures where they obtain an upper bound through product of Gaussian approximation method and a lower bound through variational approximation method. In BIBREF13, the authors combine the lower and upper bounds from approximation methods of BIBREF12 to provide a stricter bound on KL between Gaussian mixtures. Lets consider Gaussian mixtures for the words $w$ and $v$ as follows.\nThe approximate KL divergence between the Gaussian mixture representations over the words $w$ and $v$ is shown in equation DISPLAY_FORM8. More details on approximation is included in the Supplementary Material.\nwhere $EL_{ik}(w,w) = \\int f_{w,i} (\\operatorname{\\mathbf {x}}) f_{w,k} (\\operatorname{\\mathbf {x}}) d\\operatorname{\\mathbf {x}}$ and $EL_{ij}(w,v) = \\int f_{w,i} (\\operatorname{\\mathbf {x}}) f_{v,k} (\\operatorname{\\mathbf {x}}) d\\operatorname{\\mathbf {x}}$. Note that the expected likelihood kernel appears component wise inside the approximate KL divergence derivation.\nOne advantage of using KL as energy function is that it enables to capture asymmetry in entailment datasets. For eg., let us consider the words 'chair' with two senses as 'bench' and 'sling', and 'wood' with two senses as 'trees' and 'furniture'. The word chair ($w$) is entailed within wood ($v$), i.e. chair $\\models $ wood. Now, minimizing the KL divergence necessitates maximizing $\\log {\\sum _j p_{v,j} \\exp ({-KL(f_{w,i} (\\operatorname{\\mathbf {x}})||f_{v,j}(\\operatorname{\\mathbf {x}}))})}$ which in turn minimizes $KL(f_{w,i}(\\operatorname{\\mathbf {x}})||f_{v,j}(\\operatorname{\\mathbf {x}}))$. This will result in the support of the $i^{th}$ component of $w$ to be within the $j^{th}$ component of $v$, and holds for all component pairs leading to the entailment of $w$ within $v$. Consequently, we can see that bench $\\models $ trees, bench $\\models $ furniture, sling $\\models $ trees, and sling $\\models $ furniture. Thus, it introduces lexical relationship between the senses of child word and that of the parent word. Minimizing the KL also necessitates maximizing $\\log {\\sum _j {p_{v,j}} EL_{ij}(w,v)}$ term for all component pairs among $w$ and $v$. This is similar to maximizing expected likelihood kernel, which brings the means of $f_{w,i}(\\operatorname{\\mathbf {x}})$ and $f_{v,j}(\\operatorname{\\mathbf {x}})$ closer (weighted by their co-variances) as discussed in BIBREF10. Hence, the proposed approach captures the best of both worlds, thereby catering to both word similarity and entailment.\nWe also note that minimizing the KL divergence necessitates minimizing $\\log {\\sum _k p_{w,k} \\exp ({-KL(f_{w,i}||f_{w,k})})}$ which in turn maximizes $KL(f_{w,i}||f_{w,k})$. This prevents the different mixture components of a word converging to single Gaussian and encourages capturing different possible senses of the word. The same is also achieved by minimizing $\\sum _k {p_{w,k}} EL_{ik}(w,w)$ term and act as a regularization term which promotes diversity in learning senses of a word.\nExperimentation and Results\nWe train our proposed model GM$\\_$KL (Gaussian Mixture using KL Divergence) on the Text8 dataset BIBREF14 which is a pre-processed data of $17M$ words from wikipedia. Of which, 71290 unique and frequent words are chosen using the subsampling trick in BIBREF15. We compare GM$\\_$KL with the previous approaches w2g BIBREF9 ( single Gaussian model) and w2gm BIBREF10 (mixture of Gaussian model with expected likelihood kernel). For all the models used for experimentation, the embedding size ($D$) was set to 50, number of mixtures to 2, context window length to 10, batch size to 128. The word embeddings were initialized using a uniform distribution in the range of $[-\\sqrt{\\frac{3}{D}}$, $\\sqrt{\\frac{3}{D}}]$ such that the expectation of variance is 1 and mean 0 BIBREF16. One could also consider initializing the word embeddings using other contextual representations such as BERT BIBREF17 and ELMo BIBREF18 in the proposed approach. In order to purely analyze the performance of $\\emph {GM\\_KL}$ over the other models, we have chosen initialization using uniform distribution for experiments. For computational benefits, diagonal covariance is used similar to BIBREF10. Each mixture probability is constrained in the range $[0,1]$, summing to 1 by optimizing over unconstrained scores in the range $(-\\infty ,\\infty )$ and converting scores to probability using softmax function. The mixture scores are initialized to 0 to ensure fairness among all the components. The threshold for negative sampling was set to $10^{-5}$, as recommended in BIBREF3. Mini-batch gradient descent with Adagrad optimizer BIBREF19 was used with initial learning rate set to $0.05$.\nTable TABREF9 shows the qualitative results of GM$\\_$KL. Given a query word and component id, the set of nearest neighbours along with their respective component ids are listed. For eg., the word `plane' in its 0th component captures the `geometry' sense and so are its neighbours, and its 1st component captures `vehicle' sense and so are its corresponding neighbours. Other words such as `rock' captures both the `metal' and `music' senses, `star' captures `celebrity' and `astronomical' senses, and `phone' captures `telephony' and `internet' senses.\nWe quantitatively compare the performance of the GM$\\_$KL, w2g, and w2gm approaches on the SCWS dataset BIBREF6. The dataset consists of 2003 word pairs of polysemous and homonymous words with labels obtained by an average of 10 human scores. The Spearman correlation between the human scores and the model scores are computed. To obtain the model score, the following metrics are used:\nMaxCos: Maximum cosine similarity among all component pairs of words $w$ and $v$:\nAvgCos: Average component-wise cosine similarity between the words $w$ and $v$.\nKL$\\_$approx: Formulated as shown in (DISPLAY_FORM8) between the words $w$ and $v$.\nKL$\\_$comp: Maximum component-wise negative KL between words $w$ and $v$:\nTable TABREF17 compares the performance of the approaches on the SCWS dataset. It is evident from Table TABREF17 that GM$\\_$KL achieves better correlation than existing approaches for various metrics on SCWS dataset.\nTable TABREF18 shows the Spearman correlation values of GM$\\_$KL model evaluated on the benchmark word similarity datasets: SL BIBREF20, WS, WS-R, WS-S BIBREF21, MEN BIBREF22, MC BIBREF23, RG BIBREF24, YP BIBREF25, MTurk-287 and MTurk-771 BIBREF26, BIBREF27, and RW BIBREF28. The metric used for comparison is 'AvgCos'. It can be seen that for most of the datasets, GM$\\_$KL achieves significantly better correlation score than w2g and w2gm approaches. Other datasets such as MC and RW consist of only a single sense, and hence w2g model performs better and GM$\\_$KL achieves next better performance. The YP dataset have multiple senses but does not contain entailed data and hence could not make use of entailment benefits of GM$\\_$KL.\nTable TABREF19 shows the evaluation results of GM$\\_$KL model on the entailment datasets such as entailment pairs dataset BIBREF29 created from WordNet with both positive and negative labels, a crowdsourced dataset BIBREF30 of 79 semantic relations labelled as entailed or not and annotated distributionally similar nouns dataset BIBREF31. The 'MaxCos' similarity metric is used for evaluation and the best precision and best F1-score is shown, by picking the optimal threshold. Overall, GM$\\_$KL performs better than both w2g and w2gm approaches.\nConclusion\nWe proposed a KL divergence based energy function for learning multi-sense word embedding distributions modelled as Gaussian mixtures. Due to the intractability of the Gaussian mixtures for the KL divergence measure, we use an approximate KL divergence function. We also demonstrated that the proposed GM$\\_$KL approaches performed better than other approaches on the benchmark word similarity and entailment datasets.\ntocsectionAppendices\nApproximation for KL divergence between mixtures of gaussians\nKL between gaussian mixtures $f_{w}(\\operatorname{\\mathbf {x}})$ and $f_{v}(\\operatorname{\\mathbf {x}})$ can be decomposed as:\nBIBREF12 presents KL approximation between gaussian mixtures using\nproduct of gaussian approximation method where KL is approximated using product of component gaussians and\nvariational approximation method where KL is approximated by introducing some variational parameters.\nThe product of component gaussian approximation method using Jensen's inequality provides upper bounds as shown in equations DISPLAY_FORM23 and .\nThe variational approximation method provides lower bounds as shown in equations DISPLAY_FORM24 and DISPLAY_FORM25.\nwhere $H$ represents the entropy term and the entropy of $i^{th}$ component of word $w$ with dimension $D$ is given as\nIn BIBREF13, the authors combine the lower and upper bounds from approximation methods of BIBREF12 to formulate a stricter bound on KL between gaussian mixtures.\nFrom equations DISPLAY_FORM23 and DISPLAY_FORM25, a stricter lower bound for KL between gaussian mixtures is obtained as shown in equation DISPLAY_FORM26\nFrom equations and DISPLAY_FORM24, a stricter upper bound for KL between gaussian mixtures is obtained as shown in equation DISPLAY_FORM27\nFinally, the KL between gaussian mixtures is taken as the mean of KL upper and lower bounds as shown in equation DISPLAY_FORM28.",6 "char_len": 14902,7 "document_hash": "92bd83eaef8ba1d87331b4a4ffb559a76d3b5c4db13bbffaaca8a688546db7f4",8 "questions": [9 {10 "question_id": "q00",11 "qtype": "qasper",12 "question": "How does this approach compare to other WSD approaches employing word embeddings?"13 }14 ]15}