1{2 "document_id": "lb_qasper_d0c52dceef7d3b347e2f4543e601576d4d68324e7c543a6d",3 "family": "public_qasper",4 "split": "test",5 "text": "Introduction\nNatural languages evolve and words have always been subject to semantic change over time BIBREF1. With the rise of large digitized text resources recent NLP technologies have made it possible to capture such change with vector space models BIBREF2, BIBREF3, BIBREF4, BIBREF5, topic models BIBREF6, BIBREF7, BIBREF8, and sense clustering models BIBREF9. However, many approaches for detecting LSC differ profoundly from each other and therefore drawing comparisons between them can be challenging BIBREF10. Not only do architectures for detecting LSC vary, their performance is also often evaluated without access to evaluation data or too sparse data sets. In cases where evaluation data is available, oftentimes LSCD systems are not evaluated on the same data set which hinders the research community to draw comparisons.\nFor this reason we report the results of the first shared task on unsupervised lexical semantic change detection in German that is based on an annotated data set to guarantee objective reasoning throughout different approaches. The task was organized as part of the seminar 'Lexical Semantic Change Detection' at the IMS Stuttgart in the summer term of 2019.\nTask\nThe goal of the shared task was to create an architecture to detect semantic change and to rank words according to their degree of change between two different time periods. Given two corpora Ca and Cb, the target words had to be ranked according to their degree of lexical semantic change between Ca and Cb as annotated by human judges. A competition was set up on Codalab and teams mostly consisting of 2 people were formed to take part in the task. There was one group consisting of 3 team members and two individuals who entered the task on their own. In total there were 12 LSCD systems participating in the shared task.\nThe shared task was divided into three phases, i.e., development, testing and analysis phase. In the development phase each team implemented a first version of their model based on a trial data set and submitted it subsequently. In the testing phase the testing data was made public and participants applied their models to the test data with a restriction of possible result uploads to 30. The leaderboard was public at all times. Eventually, the analysis phase was entered and the models of the testing phase were evaluated in terms of the predictions they made and parameters could be tuned further. The models and results will be discussed in detail in sections 7 and 8.\nCorpora\nThe task, as framed above, requires to detect the semantic change between two corpora. The two corpora used in the shared task correspond to the diachronic corpus pair from BIBREF0: DTA18 and DTA19. They consist of subparts of DTA corpus BIBREF11 which is a freely available lemmatized, POS-tagged and spelling-normalized diachronic corpus of German containing texts from the 16th to the 20th century. DTA18 contains 26 million sentences published between 1750-1799 and DTA19 40 million between 1850-1899. The corpus version used in the task has the following format: \"year [tab] lemma1 lemma2 lemma3 ...\".\nEvaluation\nThe Diachronic Usage Relatedness (DURel) gold standard data set includes 22 target words and their varying degrees of semantic change BIBREF12. For each of these target words a random sample of use pairs from the DTA corpus was retrieved and annotated. The annotators were required to rate the pairs according to their semantic relatedness on a scale from 1 to 4 (unrelated - identical meanings) for two time periods. The average Spearman's $\\rho $ between the five annotators was 0.66 for 1,320 use paris. The resulting word ranking of the DURel data set is determined by the mean usage relatedness across two time periods and is used as the benchmark to compare the models’ performances in the shared task.\nEvaluation ::: Metric\nThe output of a system with the target words in the predicted order is compared to the gold ranking of the DURel data set. As the metric to assess how well the model's output fits the gold ranking Spearman's $\\rho $ was used. The higher Spearman's rank-order correlation the better the system's performance.\nEvaluation ::: Baselines\nModels were compared to two baselines for the shared task:\nlog-transformed normalized frequency difference (FD)\ncount vectors with column intersection and cosine distance (CNT + CI + CD)\nThe window size for CNT + CI + CD was 10. Find more information on these models in BIBREF0.\nParticipating Systems\nParticipants mostly rely on the models compared in BIBREF0 and apply modifications to improve them. In particular, most teams make use of skip-gram with negative sampling (SGNS) based on BIBREF13 to learn the semantic spaces of the two time periods and orthogonal procrustes (OP) to align these vector spaces, similar to the approach by BIBREF14. Different meaning representations such as sense clusters are used as well. As measure to detect the degree of LSC all teams except one choose cosine distance (CD). This team uses Jensen-Shannon distance (JSD) instead, which computes the distance between probability distributions BIBREF15.\nThe models of each team will be briefly introduced in this section.\nParticipating Systems ::: sorensbn\nTeam sorensbn makes use of SGNS + OP + CD to detect LSC. They use similar hyperparameters as in BIBREF0 to tune the SGNS model. They use an open-sourced noise-aware implementation to improve the OP alignment BIBREF16.\nParticipating Systems ::: tidoe\nTeam tidoe builds on SGNS + OP + CD, but they add a transformation step to receive binarized representations of matrices BIBREF17. This step is taken to counter the bias that can occur in vector-space models based on frequencies BIBREF18.\nParticipating Systems ::: in vain\nThe team applies a model based on SGNS with vector initialization alignment and cosine distance (SGNS + VI + CD). Vector initialization is an alignment strategy where the vector space learning model for $t_2$ is initialized with the vectors from $t_1$ BIBREF19. Since SGNS + VI + OP does not perform as well as other models in BIBREF0, they alter the vector initialization process by initializing on the complete model instead of only the word matrix of $t_1$ to receive improved results.\nParticipating Systems ::: Evilly\nIn line with previous approaches, team Evilly builds upon SGNS + OP + CD. They alter the OP step by using only high-frequency words for alignment.\nParticipating Systems ::: DAF\nTeam DAF uses an architecture based on learning vectors with fastText, alignment with unsupervised and supervised variations of OP, and CD, using the MUSE package BIBREF20, BIBREF21. For the supervised alignment stop words are used. The underlying assumption is that stop words serve as functional units of language and their usage should be consistent over time.\nParticipating Systems ::: SnakesOnAPlane\nThe team learns vector spaces with count vectors, positive pointwise mutual information (PPMI), SGNS and uses column intersection (CI) and OP as alignment techniques where applicable. Then they compare two distance measures (CD and JSD) for the different models CNT + CI, PPMI + CI and SGNS + OP to identify which measure performs better for these models. They also experiment with different ways to remove negative values from SGNS vectors, which is needed for JSD.\nParticipating Systems ::: TeamKulkarni15\nTeamKulkarni15 uses SGNS + OP + CD with the modification of local alignment with k nearest neighbors, since other models often use global alignment that can be prone to noise BIBREF22.\nParticipating Systems ::: Bashmaistori\nThey use word injection (WI) alignment on PPMI vectors with CD. This approach avoids the complex alignment procedure for embeddings and is applicable to embeddings and count-based methods. They compare two implementations of word injection BIBREF23, BIBREF0 as these showed different results on different data sets.\nParticipating Systems ::: giki\nTeam giki uses PPMI + CI + CD to detect LSC. They state that a word sense is determined by its context, but relevant context words can also be found outside a predefined window. Therefore, they use tf-idf to select relevant context BIBREF24.\nParticipating Systems ::: Edu-Phil\nSimilar to team DAF they also use fastText + OP + CD. Their hypothesis is that fastText may increase the performance for less frequent words in the corpus since generating word embeddings in fasttext is based on character n-grams.\nParticipating Systems ::: orangefoxes\nThey use the model by BIBREF5 which is based on SGNS, but avoids alignment by treating time as a vector that may be combined with word vectors to get time-specific word vectors.\nParticipating Systems ::: Loud Whisper\nLoud Whisper base their approach on BIBREF9 which is a graph-based sense clustering model. They process the data set to receive bigrams, create a co-occurence graph representation and after clustering assess the type of change per word by comparing the results against an intersection table. Their motivation is not only to use a graph-based approach, but to extend the approach by enabling change detection for all parts of speech as opposed to the original model.\nResults and Discussion\nTable TABREF8 shows the results of the shared task. All teams receive better results than baseline 1 (FD), of which a total of 8 teams outperform baseline 2 (CNT + CI + CD). The 4 top scores with $\\rho $ $>$ 0.7 are either modified versions of SGNS + OP + CD or use SGNS + VI + CD. The following 4 scores in the range of 0.5 $<$ $\\rho $ $<$ 0.6 are generated by the models fastText + OP + CD, SGNS + OP + CD/JSD, and PPMI + WI + CD.\nContrary to the results by BIBREF0 the modified version of vector initialization shows high performance similar to OP alignment, as previously reported by BIBREF14. Some modifications to the SGNS + OP + CD approach are able to yield better results than others, e.g. noise-aware alignment and binarized matrices as compared to frequency-driven OP alignment or local alignment with KNN. Team SnakesOnAPlane compare two distance measures and their results show that JSD ($\\rho $ $=$ .561) performs minimally worse than CD ($\\rho $ $=$ .565) as the semantic change measure for their model.\nThe overall best-performing model is Skip-Gram with orthogonal alignment and cosine distance (SGNS + OP + CD) with similar hyperparameters as in the model architecture described previously BIBREF0. Said architecture was used as the basis for the two best performing models. Team tidoe reports that binarizing matrices leads to a generally worse performance ($\\rho $ $=$ .811) compared to the unmodified version of SGNS + OP + CD ($\\rho $ $=$ 0.9). The noise aware alignment approach applied by team sorensbn obtains a higher score ($\\rho $ $=$ .854) compared to the result reported by tidoe, but is unable to exceed the performance of the unmodified SNGS + OP + CD for the same set of hyperparameters (window size = 10, negative sampling = 1; subsampling = None). Of the 8 scores above the second baseline, 5 use an architecture that builds upon SGNS + OP + CD. Whereas in the lower score segment $\\rho $ $<$ 0.5 none of the models use SGNS + OP + CD. These findings are in line with the results reported by BIBREF0, however the overall best results are lower in this shared task, which is expected from the smaller number of parameter combinations explored. Additionally, in the shared task the objective was to report the best score and not to calculate the mean which makes it more difficult to compare the robustness of the models presented here.",6 "char_len": 11559,7 "document_hash": "1f01f14721af5adaa49a460559532faa8594a2bc529c145a2617fc19ea6551d8",8 "questions": [9 {10 "question_id": "q00",11 "qtype": "qasper",12 "question": "What is the corpus used for the task?"13 }14 ]15}