{
  "document_id": "lb_qasper_168484fbdc83fc2e0c374e1d92ed08edca0c33669cd88a33",
  "family": "public_qasper",
  "split": "dev",
  "text": "Introduction\nWord Sense Disambiguation (WSD) is a fundamental task and long-standing challenge in Natural Language Processing (NLP), which aims to find the exact sense of an ambiguous word in a particular context BIBREF0. Previous WSD approaches can be grouped into two main categories: knowledge-based and supervised methods.\nKnowledge-based WSD methods rely on lexical resources like WordNet BIBREF1 and usually exploit two kinds of lexical knowledge. The gloss, which defines a word sense meaning, is first utilized in Lesk algorithm BIBREF2 and then widely taken into account in many other approaches BIBREF3, BIBREF4. Besides, structural properties of semantic graphs are mainly used in graph-based algorithms BIBREF5, BIBREF6.\nTraditional supervised WSD methods BIBREF7, BIBREF8, BIBREF9 focus on extracting manually designed features and then train a dedicated classifier (word expert) for every target lemma.\nAlthough word expert supervised WSD methods perform better, they are less flexible than knowledge-based methods in the all-words WSD task BIBREF10. Recent neural-based methods are devoted to dealing with this problem. BIBREF11 present a supervised classifier based on Bi-LSTM, which shares parameters among all word types except the last layer. BIBREF10 convert WSD task to a sequence labeling task, thus building a unified model for all polysemous words. However, neither of them can totally beat the best word expert supervised methods.\nMore recently, BIBREF12 propose to leverage the gloss information from WordNet and model the semantic relationship between the context and gloss in an improved memory network. Similarly, BIBREF13 introduce a (hierarchical) co-attention mechanism to generate co-dependent representations for the context and gloss. Their attempts prove that incorporating gloss knowledge into supervised WSD approach is helpful, but they still have not achieved much improvement, because they may not make full use of gloss knowledge.\nIn this paper, we focus on how to better leverage gloss information in a supervised neural WSD system. Recently, the pre-trained language models, such as ELMo BIBREF14 and BERT BIBREF15, have shown their effectiveness to alleviate the effort of feature engineering. Especially, BERT has achieved excellent results in question answering (QA) and natural language inference (NLI). We construct context-gloss pairs from glosses of all possible senses (in WordNet) of the target word, thus treating WSD task as a sentence-pair classification problem. We fine-tune the pre-trained BERT model and achieve new state-of-the-art results on WSD task. In particular, our contribution is two-fold:\n1. We construct context-gloss pairs and propose three BERT-based models for WSD.\n2. We fine-tune the pre-trained BERT model, and the experimental results on several English all-words WSD benchmark datasets show that our approach significantly outperforms the state-of-the-art systems.\nMethodology\nIn this section, we describe our method in detail.\nMethodology ::: Task Definition\nIn WSD, a sentence $s$ usually consists of a series of words: $\\lbrace w_1,\\cdots ,w_m\\rbrace $, and some of the words $\\lbrace w_{i_1},\\cdots ,w_{i_k}\\rbrace $ are targets $\\lbrace t_1,\\cdots ,t_k\\rbrace $ need to be disambiguated. For each target $t$, its candidate senses $\\lbrace c_1,\\cdots ,c_n\\rbrace $ come from entries of its lemma in a pre-defined sense inventory (usually WordNet). Therefore, WSD task aims to find the most suitable entry (symbolized as unique sense key) for each target in a sentence. See a sentence example in Table TABREF1.\nMethodology ::: BERT\nBERT BIBREF15 is a new language representation model, and its architecture is a multi-layer bidirectional Transformer encoder. BERT model is pre-trained on a large corpus and two novel unsupervised prediction tasks, i.e., masked language model and next sentence prediction tasks are used in pre-training. When incorporating BERT into downstream tasks, the fine-tuning procedure is recommended. We fine-tune the pre-trained BERT model on WSD task.\nMethodology ::: BERT ::: BERT(Token-CLS)\nSince every target in a sentence needs to be disambiguated to find its exact sense, WSD task can be regarded as a token-level classification task. To incorporate BERT to WSD task, we take the final hidden state of the token corresponding to the target word (if more than one token, we average them) and add a classification layer for every target lemma, which is the same as the last layer of the Bi-LSTM model BIBREF11.\nMethodology ::: GlossBERT\nBERT can explicitly model the relationship of a pair of texts, which has shown to be beneficial to many pair-wise natural language understanding tasks. In order to fully leverage gloss information, we propose GlossBERT to construct context-gloss pairs from all possible senses of the target word in WordNet, thus treating WSD task as a sentence-pair classification problem.\nWe describe our construction method with an example (See Table TABREF1). There are four targets in this sentence, and here we take target word research as an example:\nMethodology ::: GlossBERT ::: Context-Gloss Pairs\nThe sentence containing target words is denoted as context sentence. For each target word, we extract glosses of all $N$ possible senses (here $N=4$) of the target word (research) in WordNet to obtain the gloss sentence. [CLS] and [SEP] marks are added to the context-gloss pairs to make it suitable for the input of BERT model. A similar idea is also used in aspect-based sentiment analysis BIBREF16.\nMethodology ::: GlossBERT ::: Context-Gloss Pairs with Weak Supervision\nBased on the previous construction method, we add weak supervised signals to the context-gloss pairs (see the highlighted part in Table TABREF1). The signal in the gloss sentence aims to point out the target word, and the signal in the context sentence aims to emphasize the target word considering the situation that a target word may occur more than one time in the same sentence.\nTherefore, each target word has $N$ context-gloss pair training instances ($label\\in \\lbrace yes, no\\rbrace $). When testing, we output the probability of $label=yes$ of each context-gloss pair and choose the sense corresponding to the highest probability as the prediction label of the target word. We experiment with three GlossBERT models:\nMethodology ::: GlossBERT ::: GlossBERT(Token-CLS)\nWe use context-gloss pairs as input. We highlight the target word by taking the final hidden state of the token corresponding to the target word (if more than one token, we average them) and add a classification layer ($label\\in \\lbrace yes, no\\rbrace $).\nMethodology ::: GlossBERT ::: GlossBERT(Sent-CLS)\nWe use context-gloss pairs as input. We take the final hidden state of the first token [CLS] as the representation of the whole sequence and add a classification layer ($label\\in \\lbrace yes, no\\rbrace $), which does not highlight the target word.\nMethodology ::: GlossBERT ::: GlossBERT(Sent-CLS-WS)\nWe use context-gloss pairs with weak supervision as input. We take the final hidden state of the first token [CLS] and add a classification layer ($label\\in \\lbrace yes, no\\rbrace $), which weekly highlight the target word by the weak supervision.\nExperiments ::: Datasets\nThe statistics of the WSD datasets are shown in Table TABREF12.\nExperiments ::: Datasets ::: Training Dataset\nFollowing previous work BIBREF13, BIBREF12, BIBREF10, BIBREF17, BIBREF9, BIBREF7, we choose SemCor3.0 as training corpus, which is the largest corpus manually annotated with WordNet sense for WSD.\nExperiments ::: Datasets ::: Evaluation Datasets\nWe evaluate our method on several English all-words WSD datasets. For a fair comparison, we use the benchmark datasets proposed by BIBREF17 which include five standard all-words fine-grained WSD datasets from the Senseval and SemEval competitions: Senseval-2 (SE2), Senseval-3 (SE3), SemEval-2007 (SE07), SemEval-2013 (SE13) and SemEval-2015 (SE15). Following BIBREF13, BIBREF12 and BIBREF10, we choose SE07, the smallest among these test sets, as the development set.\nExperiments ::: Datasets ::: WordNet\nSince BIBREF17 map all the sense annotations in these datasets from their original versions to WordNet 3.0, we extract word sense glosses from WordNet 3.0.\nExperiments ::: Settings\nWe use the pre-trained uncased BERT$_\\mathrm {BASE}$ model for fine-tuning, because we find that BERT$_\\mathrm {LARGE}$ model performs slightly worse than BERT$_\\mathrm {BASE}$ in this task. The number of Transformer blocks is 12, the number of the hidden layer is 768, the number of self-attention heads is 12, and the total number of parameters of the pre-trained model is 110M. When fine-tuning, we use the development set (SE07) to find the optimal settings for our experiments. We keep the dropout probability at 0.1, set the number of epochs to 4. The initial learning rate is 2e-5, and the batch size is 64.\nExperiments ::: Results\nTable TABREF19 shows the performance of our method on the English all-words WSD benchmark datasets. We compare our approach with previous methods.\nThe first block shows the MFS baseline, which selects the most frequent sense in the training corpus for each target word.\nThe second block shows two knowledge-based systems. Lesk$_{ext+emb}$ BIBREF4 is a variant of Lesk algorithm BIBREF2 by calculating the gloss-context overlap of the target word. Babelfy BIBREF6 is a unified graph-based approach which exploits the semantic network structure from BabelNet.\nThe third block shows two word expert traditional supervised systems. IMS BIBREF7 is a flexible framework which trains SVM classifiers and uses local features. And IMS$_{+emb}$ BIBREF9 is the best configuration of the IMS framework, which also integrates word embeddings as features.\nThe fourth block shows several recent neural-based methods. Bi-LSTM BIBREF11 is a baseline for neural models. Bi-LSTM$_{+ att. + LEX + POS}$ BIBREF10 is a multi-task learning framework for WSD, POS tagging, and LEX with self-attention mechanism, which converts WSD to a sequence learning task. GAS$_{ext}$ BIBREF12 is a variant of GAS which is a gloss-augmented variant of the memory network by extending gloss knowledge. CAN$^s$ and HCAN BIBREF13 are sentence-level and hierarchical co-attention neural network models which leverage gloss knowledge.\nIn the last block, we report the performance of our method. BERT(Token-CLS) is our baseline, which does not incorporate gloss information, and it performs slightly worse than previous traditional supervised methods and recent neural-based methods. It proves that directly using BERT cannot obtain performance growth. The other three methods outperform other models by a substantial margin, which proves that the improvements come from leveraging BERT to better exploit gloss information. It is worth noting that our method achieves significant improvements in SE07 and Verb over previous methods, which have the highest ambiguity level among all datasets and all POS tags respectively according to BIBREF17.\nMoreover, GlossBERT(Token-CLS) performs better than GlossBERT(Sent-CLS), which proves that highlighting the target word in the sentence is important. However, the weakly highlighting method GlossBERT(Sent-CLS-WS) performs best in most circumstances, which may result from its combination of the advantages of the other two methods.\nExperiments ::: Discussion\nThere are two main reasons for the great improvements of our experimental results. First, we construct context-gloss pairs and convert WSD problem to a sentence-pair classification task which is similar to NLI tasks and train only one classifier, which is equivalent to expanding the corpus. Second, we leverage BERT BIBREF15 to better exploit the gloss information. BERT model shows its advantage in dealing with sentence-pair classification tasks by its amazing improvement on QA and NLI tasks. This advantage comes from both of its two novel unsupervised prediction tasks.\nCompared with traditional word expert supervised methods, our GlossBERT shows its effectiveness to alleviate the effort of feature engineering and does not require training a dedicated classifier for every target lemma. Up to now, it can be said that the neural network method can totally beat the traditional word expert method. Compared with recent neural-based methods, our solution is more intuitive and can make better use of gloss knowledge. Besides, our approach demonstrates that when we fine-tune BERT on a downstream task, converting it into a sentence-pair classification task may be a good choice.\nConclusion\nIn this paper, we seek to better leverage gloss knowledge in a supervised neural WSD system. We propose a new solution to WSD by constructing context-gloss pairs and then converting WSD to a sentence-pair classification task. We fine-tune the pre-trained BERT model and achieve new state-of-the-art results on WSD task.\nAcknowledgments\nWe would like to thank the anonymous reviewers for their valuable comments. The research work is supported by National Natural Science Foundation of China (No. 61751201 and 61672162), Shanghai Municipal Science and Technology Commission (16JC1420401 and 17JC1404100), Shanghai Municipal Science and Technology Major Project (No.2018SHZDZX01) and ZJLab.",
  "char_len": 13298,
  "document_hash": "0cc8f73bdbf9b91d943f36b3c59551c0a194bcfe10bb303e11356a4e8d7969c3",
  "questions": [
    {
      "question_id": "q00",
      "qtype": "qasper",
      "question": "Do they use large or small BERT?"
    }
  ]
}