1{2 "document_id": "lb_qasper_d17f6335089f3dd922f3d5400e3fc2ad1ca15202f4ac6467",3 "family": "public_qasper",4 "split": "dev",5 "text": "Introduction\nTwitter, a micro-blogging and social networking site has emerged as a platform where people express themselves and react to events in real-time. It is estimated that nearly 500 million tweets are sent per day . Twitter data is particularly interesting because of its peculiar nature where people convey messages in short sentences using hashtags, emoticons, emojis etc. In addition, each tweet has meta data like location and language used by the sender. It's challenging to analyze this data because the tweets might not be grammatically correct and the users tend to use informal and slang words all the time. Hence, this poses an interesting problem for NLP researchers. Any advances in using this abundant and diverse data can help understand and analyze information about a person, an event, a product, an organization or a country as a whole. Many notable use cases of the twitter can be found here.\nAlong the similar lines, The Task 1 of WASSA-2017 BIBREF0 poses a problem of finding emotion intensity of four emotions namely anger, fear, joy, sadness from tweets. In this paper, we describe our approach and experiments to solve this problem. The rest of the paper is laid out as follows: Section 2 describes the system architecture, Section 3 reports results and inference from different experiments, while Section 4 points to ways that the problem can be further explored.\nPreprocessing\nThe preprocessing step modifies the raw tweets before they are passed to feature extraction. Tweets are processed using tweetokenize tool. Twitter specific features are replaced as follows: username handles to USERNAME, phone numbers to PHONENUMBER, numbers to NUMBER, URLs to URL and times to TIME. A continuous sequence of emojis is broken into individual tokens. Finally, all tokens are converted to lowercase.\nFeature Extraction\nMany tasks related to sentiment or emotion analysis depend upon affect, opinion, sentiment, sense and emotion lexicons. These lexicons associate words to corresponding sentiment or emotion metrics. On the other hand, the semantic meaning of words, sentences, and documents are preserved and compactly represented using low dimensional vectors BIBREF1 instead of one hot encoding vectors which are sparse and high dimensional. Finally, there are traditional NLP features like word N-grams, character N-grams, Part-Of-Speech N-grams and word clusters which are known to perform well on various tasks.\nBased on these observations, the feature extraction step is implemented as a union of different independent feature extractors (featurizers) in a light-weight and easy to use Python program EmoInt . It comprises of all features available in the baseline model BIBREF2 along with additional feature extractors and bi-gram support. Fourteen such feature extractors have been implemented which can be clubbed into 3 major categories:\n[noitemsep]\nLexicon Features\nWord Vectors\nSyntax Features\nLexicon Features: AFINN BIBREF3 word list are manually rated for valence with an integer between -5 (Negative Sentiment) and +5 (Positive Sentiment). Bing Liu BIBREF4 opinion lexicon extract opinion on customer reviews. +/-EffectWordNet BIBREF5 by MPQA group are sense level lexicons. The NRC Affect Intensity BIBREF6 lexicons provide real valued affect intensity. NRC Word-Emotion Association Lexicon BIBREF7 contains 8 sense level associations (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and 2 sentiment level associations (negative and positive). Expanded NRC Word-Emotion Association Lexicon BIBREF8 expands the NRC word-emotion association lexicon for twitter specific language. NRC Hashtag Emotion Lexicon BIBREF9 contains emotion word associations computed on emotion labeled twitter corpus via Hashtags. NRC Hashtag Sentiment Lexicon and Sentiment140 Lexicon BIBREF10 contains sentiment word associations computed on twitter corpus via Hashtags and Emoticons. SentiWordNet BIBREF11 assigns to each synset of WordNet three sentiment scores: positivity, negativity, objectivity. Negation lexicons collections are used to count the total occurrence of negative words. In addition to these, SentiStrength BIBREF12 application which estimates the strength of positive and negative sentiment from tweets is also added.\nWord Vectors: We focus primarily on the word vector representations (word embeddings) created specifically using the twitter dataset. GloVe BIBREF13 is an unsupervised learning algorithm for obtaining vector representations for words. 200-dimensional GloVe embeddings trained on 2 Billion tweets are integrated. Edinburgh embeddings BIBREF14 are obtained by training skip-gram model on Edinburgh corpus BIBREF15 . Since tweets are abundant with emojis, Emoji embeddings BIBREF16 which are learned from the emoji descriptions have been used. Embeddings for each tweet are obtained by summing up individual word vectors and then dividing by the number of tokens in the tweet.\nSyntactic Features: Syntax specific features such as Word N-grams, Part-Of-Speech N-grams BIBREF17 , Brown Cluster N-grams BIBREF18 obtained using TweetNLP project have been integrated into the system.\nThe final feature vector is the concatenation of all the individual features. For example, we concatenate average word vectors, sum of NRC Affect Intensities, number of positive and negative Bing Liu lexicons, number of negation words and so on to get final feature vector. The scaling of final features is not required when used with gradient boosted trees. However, scaling steps like standard scaling (zero mean and unit normal) may be beneficial for neural networks as the optimizers work well when the data is centered around origin.\nA total of fourteen different feature extractors have been implemented, all of which can be enabled or disabled individually to extract features from a given tweet.\nRegression\nThe dev data set BIBREF19 in the competition was small hence, the train and dev sets were merged to perform 10-fold cross validation. On each fold, a model was trained and the predictions were collected on the remaining dataset. The predictions are averaged across all the folds to generalize the solution and prevent over-fitting. As described in Section SECREF6 , different combinations of feature extractors were used. After performing feature extraction, the data was then passed to various regressors Support Vector Regression, AdaBoost, RandomForestRegressor, and, BaggingRegressor of sklearn BIBREF20 . Finally, the chosen top performing models had the least error on evaluation metrics namely Pearson's Correlation Coefficient and Spearman's rank-order correlation.\nParameter Optimization\nIn order to find the optimal parameter values for the EmoInt system, an extensive grid search was performed through the scikit-Learn framework over all subsets of the training set (shuffled), using stratified 10-fold cross validation and optimizing the Pearson's Correlation score. Best cross-validation results were obtained using AdaBoost meta regressor with base regressor as XGBoost BIBREF21 with 1000 estimators and 0.1 learning rate. Experiments and analysis of results are presented in the next section.\nExperimental Results\nAs described in Section SECREF6 various syntax features were used namely, Part-of-Speech tags, brown clusters of TweetNLP project. However, these didn't perform well in cross validation. Hence, they were dropped from the final system. While performing grid-search as mentioned in Section SECREF14 , keeping all the lexicon based features same, choice of combination of emoji vector and word vectors are varied to minimize cross validation metric. Table TABREF16 describes the results for experiments conducted with different combinations of word vectors. Emoji embeddings BIBREF16 give better results than using plain GloVe and Edinburgh embeddings. Edinburgh embeddings outperform GloVe embeddings in Joy and Sadness category but lag behind in Anger and Fear category. The official submission comprised of the top-performing model for each emotion category. This system ranked 3 for the entire test dataset and 2 for the subset of the test data formed by taking every instance with a gold emotion intensity score greater than or equal to 0.5. Post competition, experiments were performed on ensembling diverse models for improving the accuracy. An ensemble obtained by averaging the results of the top 2 performing models outperforms all the individual models.\nFeature Importance\nThe relative feature importance can be assessed by the relative depth of the feature used as a decision node in the tree. Features used at the top of the tree contribute to the final prediction decision of a larger fraction of the input samples. The expected fraction of the samples they contribute to can thus be used as an estimate of the relative importance of the features. By averaging the measure over several randomized trees, the variance of the estimate can be reduced and used as a measure of relative feature importance. In Figure FIGREF18 feature importance graphs are plotted for each emotion to infer which features are playing the major role in identifying emotional intensity in tweets. +/-EffectWordNet BIBREF5 , NRC Hashtag Sentiment Lexicon, Sentiment140 Lexicon BIBREF10 and NRC Hashtag Emotion Lexicon BIBREF9 are playing the most important role.\nSystem Limitations\nIt is important to understand how the model performs in different scenarios. Table TABREF20 analyzes when the system performs the best and worst for each emotion. Since the features used are mostly lexicon based, the system has difficulties in capturing the overall sentiment and it leads to amplifying or vanishing intensity signals. For instance, in example 4 of fear louder and shaking lexicons imply fear but overall sentence doesn't imply fear. A similar pattern can be found in the 4 example of Anger and 3 example of Joy. The system has difficulties in understanding of sarcastic tweets, for instance, in the 3 tweet of Anger the user expressed anger but used lol which is used in a positive sense most of the times and hence the system did a bad job at predicting intensity. The system also fails in predicting sentences having deeper emotion and sentiment which humans can understand with a little context. For example, in sample 4 of sadness, the tweet refers to post travel blues which humans can understand. But with little context, it is difficult for the system to accurately estimate the intensity. The performance is poor with very short sentences as there are fewer indicators to provide a reasonable estimate.\nFuture Work & Conclusion\nThe paper studies the effectiveness of various affect lexicons word embeddings to estimate emotional intensity in tweets. A light-weight easy to use affect computing framework (EmoInt) to facilitate ease of experimenting with various lexicon features for text tasks is open-sourced. It provides plug and play access to various feature extractors and handy scripts for creating ensembles.\nFew problems explained in the analysis section can be resolved with the help of sentence embeddings which take the context information into consideration. The features used in the system are generic enough to use them in other affective computing tasks on social media text, not just tweet data. Another interesting feature of lexicon-based systems is their good run-time performance during prediction, future work to benchmark the performance of the system can prove vital for deploying in a real-world setting.\nAcknowledgement\nWe would like to thank the organizers of the WASSA-2017 Shared Task on Emotion Intensity, for providing the data, the guidelines and timely support.",6 "char_len": 11686,7 "document_hash": "65e68a19e9e6fdea15d7a7b83f5ebf88cf25d58b1e2c55483669fa0806d104d1",8 "questions": [9 {10 "question_id": "q00",11 "qtype": "qasper",12 "question": "what pretrained word embeddings were used?"13 }14 ]15}