{
  "document_id": "lb_qasper_3278efeb1fb7f569dfd6ef8627cdefb5955b93b612fe2902",
  "family": "public_qasper",
  "split": "dev",
  "text": "Introduction\nA hashtag is a keyphrase represented as a sequence of alphanumeric characters plus underscore, preceded by the # symbol. Hashtags play a central role in online communication by providing a tool to categorize the millions of posts generated daily on Twitter, Instagram, etc. They are useful in search, tracking content about a certain topic BIBREF0 , BIBREF1 , or discovering emerging trends BIBREF2 .\nHashtags often carry very important information, such as emotion BIBREF3 , sentiment BIBREF4 , sarcasm BIBREF5 , and named entities BIBREF6 , BIBREF7 . However, inferring the semantics of hashtags is non-trivial since many hashtags contain multiple tokens joined together, which frequently leads to multiple potential interpretations (e.g., lion head vs. lionhead). Table TABREF3 shows several examples of single- and multi-token hashtags. While most hashtags represent a mix of standard tokens, named entities and event names are prevalent and pose challenges to both human and automatic comprehension, as these are more likely to be rare tokens. Hashtags also tend to be shorter to allow fast typing, to attract attention or to satisfy length limitations imposed by some social media platforms. Thus, they tend to contain a large number of abbreviations or non-standard spelling variations (e.g., #iloveu4eva) BIBREF8 , BIBREF9 , which hinders their understanding.\nThe goal of our study is to build efficient methods for automatically splitting a hashtag into a meaningful word sequence. Our contributions are:\nOur new dataset includes segmentation for 12,594 unique hashtags and their associated tweets annotated in a multi-step process for higher quality than the previous dataset of 1,108 hashtags BIBREF10 . We frame the segmentation task as a pairwise ranking problem, given a set of candidate segmentations. We build several neural architectures using this problem formulation which use corpus-based, linguistic and thesaurus based features. We further propose a multi-task learning approach which jointly learns segment ranking and single- vs. multi-token hashtag classification. The latter leads to an error reduction of 24.6% over the current state-of-the-art. Finally, we demonstrate the utility of our method by using hashtag segmentation in the downstream task of sentiment analysis. Feeding the automatically segmented hashtags to a state-of-the-art sentiment analysis method on the SemEval 2017 benchmark dataset results in a 2.6% increase in the official metric for the task.\nBackground and Preliminaries\nCurrent approaches for hashtag segmentation can be broadly divided into three categories: (a) gazeteer and rule based BIBREF11 , BIBREF12 , BIBREF13 , (b) word boundary detection BIBREF14 , BIBREF15 , and (c) ranking with language model and other features BIBREF16 , BIBREF10 , BIBREF0 , BIBREF17 , BIBREF18 . Hashtag segmentation approaches draw upon work on compound splitting for languages such as German or Finnish BIBREF19 and word segmentation BIBREF20 for languages with no spaces between words such as Chinese BIBREF21 , BIBREF22 . Similar to our work, BIBREF10 BansalBV15 extract an initial set of candidate segmentations using a sliding window, then rerank them using a linear regression model trained on lexical, bigram and other corpus-based features. The current state-of-the-art approach BIBREF14 , BIBREF15 uses maximum entropy and CRF models with a combination of language model and hand-crafted features to predict if each character in the hashtag is the beginning of a new word.\nGenerating Candidate Segmentations. Microsoft Word Breaker BIBREF16 is, among the existing methods, a strong baseline for hashtag segmentation, as reported in BIBREF14 and BIBREF10 . It employs a beam search algorithm to extract INLINEFORM0 best segmentations as ranked by the n-gram language model probability: INLINEFORM1\nwhere INLINEFORM0 is the word sequence of segmentation INLINEFORM1 and INLINEFORM2 is the window size. More sophisticated ranking strategies, such as Binomial and word length distribution based ranking, did not lead to a further improvement in performance BIBREF16 . The original Word Breaker was designed for segmenting URLs using language models trained on web data. In this paper, we reimplemented and tailored this approach to segmenting hashtags by using a language model specifically trained on Twitter data (implementation details in § SECREF26 ). The performance of this method itself is competitive with state-of-the-art methods (evaluation results in § SECREF46 ). Our proposed pairwise ranking method will effectively take the top INLINEFORM3 segmentations generated by this baseline as candidates for reranking.\nHowever, in prior work, the ranking scores of each segmentation were calculated independently, ignoring the relative order among the top INLINEFORM0 candidate segmentations. To address this limitation, we utilize a pairwise ranking strategy for the first time for this task and propose neural architectures to model this.\nMulti-task Pairwise Neural Ranking\nWe propose a multi-task pairwise neural ranking approach to better incorporate and distinguish the relative order between the candidate segmentations of a given hashtag. Our model adapts to address single- and multi-token hashtags differently via a multi-task learning strategy without requiring additional annotations. In this section, we describe the task setup and three variants of pairwise neural ranking models (Figure FIGREF11 ).\nSegmentation as Pairwise Ranking\nThe goal of hashtag segmentation is to divide a given hashtag INLINEFORM0 into a sequence of meaningful words INLINEFORM1 . For a hashtag of INLINEFORM2 characters, there are a total of INLINEFORM3 possible segmentations but only one, or occasionally two, of them ( INLINEFORM4 ) are considered correct (Table TABREF9 ).\nWe transform this task into a pairwise ranking problem: given INLINEFORM0 candidate segmentations { INLINEFORM1 }, we rank them by comparing each with the rest in a pairwise manner. More specifically, we train a model to predict a real number INLINEFORM2 for any two candidate segmentations INLINEFORM3 and INLINEFORM4 of hashtag INLINEFORM5 , which indicates INLINEFORM6 is a better segmentation than INLINEFORM7 if positive, and vice versa. To quantify the quality of a segmentation in training, we define a gold scoring function INLINEFORM8 based on the similarities with the ground-truth segmentation INLINEFORM9 : INLINEFORM10\nWe use the Levenshtein distance (minimum number of single-character edits) in this paper, although it is possible to use other similarity measurements as alternatives. We use the top INLINEFORM0 segmentations generated by Microsoft Word Breaker (§ SECREF2 ) as initial candidates.\nPairwise Neural Ranking Model\nFor an input candidate segmentation pair INLINEFORM0 , we concatenate their feature vectors INLINEFORM1 and INLINEFORM2 , and feed them into a feedforward network which emits a comparison score INLINEFORM3 . The feature vector INLINEFORM4 or INLINEFORM5 consists of language model probabilities using Good-Turing BIBREF23 and modified Kneser-Ney smoothing BIBREF24 , BIBREF25 , lexical and linguistic features (more details in § SECREF23 ). For training, we use all the possible pairs INLINEFORM6 of the INLINEFORM7 candidates as the input and their gold scores INLINEFORM8 as the target. The training objective is to minimize the Mean Squared Error (MSE): DISPLAYFORM0\nwhere INLINEFORM0 is the number of training examples.\nTo aggregate the pairwise comparisons, we follow a greedy algorithm proposed by BIBREF26 cohen1998learning and used for preference ranking BIBREF27 . For each segmentation INLINEFORM0 in the candidate set INLINEFORM1 , we calculate a single score INLINEFORM2 , and find the segmentation INLINEFORM3 corresponding to the highest score. We repeat the same procedure after removing INLINEFORM4 from INLINEFORM5 , and continue until INLINEFORM6 reduces to an empty set. Figure FIGREF11 (a) shows the architecture of this model.\nMargin Ranking (MR) Loss\nAs an alternative to the pairwise ranker (§ SECREF15 ), we propose a pairwise model which learns from candidate pairs INLINEFORM0 but ranks each individual candidate directly rather than relatively. We define a new scoring function INLINEFORM1 which assigns a higher score to the better candidate, i.e., INLINEFORM2 , if INLINEFORM3 is a better candidate than INLINEFORM4 and vice-versa. Instead of concatenating the features vectors INLINEFORM5 and INLINEFORM6 , we feed them separately into two identical feedforward networks with shared parameters. During testing, we use only one of the networks to rank the candidates based on the INLINEFORM7 scores. For training, we add a ranking layer on top of the networks to measure the violations in the ranking order and minimize the Margin Ranking Loss (MR): DISPLAYFORM0\nwhere INLINEFORM0 is the number of training samples. The architecture of this model is presented in Figure FIGREF11 (b).\nAdaptive Multi-task Learning\nBoth models in § SECREF15 and § SECREF17 treat all the hashtags uniformly. However, different features address different types of hashtags. By design, the linguistic features capture named entities and multi-word hashtags that exhibit word shape patterns, such as camel case. The ngram probabilities with Good-Turing smoothing gravitate towards multi-word segmentations with known words, as its estimate for unseen ngrams depends on the fraction of ngrams seen once which can be very low BIBREF28 . The modified Kneser-Ney smoothing is more likely to favor segmentations that contain rare words, and single-word segmentations in particular. Please refer to § SECREF46 for a more detailed quantitative and qualitative analysis.\nTo leverage this intuition, we introduce a binary classification task to help the model differentiate single-word from multi-word hashtags. The binary classifier takes hashtag features INLINEFORM0 as the input and outputs INLINEFORM1 , which represents the probability of INLINEFORM2 being a multi-word hashtag. INLINEFORM3 is used as an adaptive gating value in our multi-task learning setup. The gold labels for this task are obtained at no extra cost by simply verifying whether the ground-truth segmentation has multiple words. We train the pairwise segmentation ranker and the binary single- vs. multi-token hashtag classifier jointly, by minimizing INLINEFORM4 for the pairwise ranker and the Binary Cross Entropy Error ( INLINEFORM5 ) for the classifier: DISPLAYFORM0\nwhere INLINEFORM0 is the adaptive gating value, INLINEFORM1 indicates if INLINEFORM2 is actually a multi-word hashtag and INLINEFORM3 is the number of training examples. INLINEFORM4 and INLINEFORM5 are the weights for each loss. For our experiments, we apply equal weights.\nMore specifically, we divide the segmentation feature vector INLINEFORM0 into two subsets: (a) INLINEFORM1 with modified Kneser-Ney smoothing features, and (b) INLINEFORM2 with Good-Turing smoothing and linguistic features. For an input candidate segmentation pair INLINEFORM3 , we construct two pairwise vectors INLINEFORM4 and INLINEFORM5 by concatenation, then combine them based on the adaptive gating value INLINEFORM6 before feeding them into the feedforward network INLINEFORM7 for pairwise ranking: DISPLAYFORM0\nWe use summation with padding, as we find this simple ensemble method achieves similar performance in our experiments as the more complex multi-column networks BIBREF29 . Figure FIGREF11 (c) shows the architecture of this model. An analogue multi-task formulation can also be used for the Margin Ranking loss as: DISPLAYFORM0\nFeatures\nWe use a combination of corpus-based and linguistic features to rank the segmentations. For a candidate segmentation INLINEFORM0 , its feature vector INLINEFORM1 includes the number of words in the candidate, the length of each word, the proportion of words in an English dictionary or Urban Dictionary BIBREF30 , ngram counts from Google Web 1TB corpus BIBREF31 , and ngram probabilities from trigram language models trained on the Gigaword corpus BIBREF32 and 1.1 billion English tweets from 2010, respectively. We train two language models on each corpus: one with Good-Turing smoothing using SRILM BIBREF33 and the other with modified Kneser-Ney smoothing using KenLM BIBREF34 . We also add boolean features, such as if the candidate is a named-entity present in the list of Wikipedia titles, and if the candidate segmentation INLINEFORM2 and its corresponding hashtag INLINEFORM3 satisfy certain word-shapes (more details in appendix SECREF61 ).\nSimilarly, for hashtag INLINEFORM0 , we extract the feature vector INLINEFORM1 consisting of hashtag length, ngram count of the hashtag in Google 1TB corpus BIBREF31 , and boolean features indicating if the hashtag is in an English dictionary or Urban Dictionary, is a named-entity, is in camel case, ends with a number, and has all the letters as consonants. We also include features of the best-ranked candidate by the Word Breaker model.\nImplementation Details\nWe use the PyTorch framework to implement our multi-task pairwise ranking model. The pairwise ranker consists of an input layer, three hidden layers with eight nodes in each layer and hyperbolic tangent ( INLINEFORM0 ) activation, and a single linear output node. The auxiliary classifier consists of an input layer, one hidden layer with eight nodes and one output node with sigmoid activation. We use the Adam algorithm BIBREF35 for optimization and apply a dropout of 0.5 to prevent overfitting. We set the learning rate to 0.01 and 0.05 for the pairwise ranker and auxiliary classifier respectively. For each experiment, we report results obtained after 100 epochs.\nFor the baseline model used to extract the INLINEFORM0 initial candidates, we reimplementated the Word Breaker BIBREF16 as described in § SECREF2 and adapted it to use a language model trained on 1.1 billion tweets with Good-Turing smoothing using SRILM BIBREF33 to give a better performance in segmenting hashtags (§ SECREF46 ). For all our experiments, we set INLINEFORM1 .\nHashtag Segmentation Data\nWe use two datasets for experiments (Table TABREF29 ): (a) STAN INLINEFORM0 , created by BIBREF10 BansalBV15, which consists of 1,108 unique English hashtags from 1,268 randomly selected tweets in the Stanford Sentiment Analysis Dataset BIBREF36 along with their crowdsourced segmentations and our additional corrections; and (b) STAN INLINEFORM1 , our new expert curated dataset, which includes all 12,594 unique English hashtags and their associated tweets from the same Stanford dataset.\nExperiments\nIn this section, we present experimental results that compare our proposed method with the other state-of-the-art approaches on hashtag segmentation datasets. The next section will show experiments of applying hashtag segmentation to the popular task of sentiment analysis.\nExisting Methods\nWe compare our pairwise neural ranker with the following baseline and state-of-the-art approaches:\nThe original hashtag as a single token;\nA rule-based segmenter, which employs a set of word-shape rules with an English dictionary BIBREF13 ;\nA Viterbi model which uses word frequencies from a book corpus BIBREF0 ;\nThe specially developed GATE Hashtag Tokenizer from the open source toolkit, which combines dictionaries and gazetteers in a Viterbi-like algorithm BIBREF11 ;\nA maximum entropy classifier (MaxEnt) trained on the STAN INLINEFORM0 training dataset. It predicts whether a space should be inserted at each position in the hashtag and is the current state-of-the-art BIBREF14 ;\nOur reimplementation of the Word Breaker algorithm which uses beam search and a Twitter ngram language model BIBREF16 ;\nA pairwise linear ranker which we implemented for comparison purposes with the same features as our neural model, but using perceptron as the underlying classifier BIBREF38 and minimizing the hinge loss between INLINEFORM0 and a scoring function similar to INLINEFORM1 . It is trained on the STAN INLINEFORM2 dataset.\nEvaluation Metrics\nWe evaluate the performance by the top INLINEFORM0 ( INLINEFORM1 ) accuracy (A@1, A@2), average token-level F INLINEFORM2 score (F INLINEFORM3 @1), and mean reciprocal rank (MRR). In particular, the accuracy and MRR are calculated at the segmentation-level, which means that an output segmentation is considered correct if and only if it fully matches the human segmentation. Average token-level F INLINEFORM4 score accounts for partially correct segmentation in the multi-token hashtag cases.\nResults\nTables TABREF32 and TABREF33 show the results on the STAN INLINEFORM0 and STAN INLINEFORM1 datasets, respectively. All of our pairwise neural rankers are trained on the 2,518 manually segmented hashtags in the training set of STAN INLINEFORM2 and perform favorably against other state-of-the-art approaches. Our best model (MSE+multitask) that utilizes different features adaptively via a multi-task learning procedure is shown to perform better than simply combining all the features together (MR and MSE). We highlight the 24.6% error reduction on STAN INLINEFORM3 and 16.5% on STAN INLINEFORM4 of our approach over the previous SOTA BIBREF14 on the Multi-token hashtags, and the importance of having a separate evaluation of multi-word cases as it is trivial to obtain 100% accuracy for Single-token hashtags. While our hashtag segmentation model is achieving a very high accuracy@2, to be practically useful, it remains a challenge to get the top one predication exactly correct. Some hashtags are very difficult to interpret, e.g., #BTVbrownSMB refers to the Social Media Breakfast (SMB) in Burlington, Vermont (BTV).\nThe improved Word Breaker with our addition of a Twitter-specific language model is a very strong baseline, which echos the findings of the original Word Breaker paper BIBREF16 that having a large in-domain language model is extremely helpful for word segmentation tasks. It is worth noting that the other state-of-the-art system BIBREF14 also utilized a 4-gram language model trained on 476 million tweets from 2009.\nAnalysis and Discussion\nTo empirically illustrate the effectiveness of different features on different types of hashtags, we show the results for models using individual feature sets in pairwise ranking models (MSE) in Table TABREF45 . Language models with modified Kneser-Ney smoothing perform best on single-token hashtags, while Good-Turing and Linguistic features work best on multi-token hashtags, confirming our intuition about their usefulness in a multi-task learning approach. Table TABREF47 shows a qualitative analysis with the first column ( INLINEFORM0 INLINEFORM1 INLINEFORM2 ) indicating which features lead to correct or wrong segmentations, their count in our data and illustrative examples with human segmentation.\nAs expected, longer hashtags with more than three tokens pose greater challenges and the segmentation-level accuracy of our best model (MSE+multitask) drops to 82.1%. For many error cases, our model predicts a close-to-correct segmentation, e.g., #youbrownknowyoubrownupttoobrownearly, #iseebrownlondoniseebrownfrance, which is also reflected by the higher token-level F INLINEFORM0 scores across hashtags with different lengths (Figure FIGREF51 ).\nSince our approach heavily relies on building a Twitter language model, we experimented with its sizes and show the results in Figure FIGREF52 . Our approach can perform well even with access to a smaller amount of tweets. The drop in F INLINEFORM0 score for our pairwise neural ranker is only 1.4% and 3.9% when using the language models trained on 10% and 1% of the total 1.1 billion tweets, respectively.\nLanguage use in Twitter changes with time BIBREF9 . Our pairwise ranker uses language models trained on the tweets from the year 2010. We tested our approach on a set of 500 random English hashtags posted in tweets from the year 2019 and show the results in Table TABREF55 . With a segmentation-level accuracy of 94.6% and average token-level F INLINEFORM0 score of 95.6%, our approach performs favorably on 2019 hashtags.\nExtrinsic Evaluation: Twitter Sentiment Analysis\nWe attempt to demonstrate the effectiveness of our hashtag segmentation system by studying its impact on the task of sentiment analysis in Twitter BIBREF39 , BIBREF40 , BIBREF41 . We use our best model (MSE+multitask), under the name HashtagMaster, in the following experiments.\nExperimental Setup\nWe compare the performance of the BiLSTM+Lex BIBREF42 sentiment analysis model under three configurations: (a) tweets with hashtags removed, (b) tweets with hashtags as single tokens excluding the # symbol, and (c) tweets with hashtags as segmented by our system, HashtagMaster. BiLSTM+Lex is a state-of-the-art open source system for predicting tweet-level sentiment BIBREF43 . It learns a context-sensitive sentiment intensity score by leveraging a Twitter-based sentiment lexicon BIBREF44 . We use the same settings as described by BIBREF42 teng-vo-zhang:2016:EMNLP2016 to train the model.\nWe use the dataset from the Sentiment Analysis in Twitter shared task (subtask A) at SemEval 2017 BIBREF41 . Given a tweet, the goal is to predict whether it expresses POSITIVE, NEGATIVE or NEUTRAL sentiment. The training and development sets consist of 49,669 tweets and we use 40,000 for training and the rest for development. There are a total of 12,284 tweets containing 12,128 hashtags in the SemEval 2017 test set, and our hashtag segmenter ended up splitting 6,975 of those hashtags present in 3,384 tweets.\nResults and Analysis\nIn Table TABREF59 , we report the results based on the 3,384 tweets where HashtagMaster predicted a split, as for the rest of tweets in the test set, the hashtag segmenter would neither improve nor worsen the sentiment prediction. Our hashtag segmenter successfully improved the sentiment analysis performance by 2% on average recall and F INLINEFORM0 comparing to having hashtags unsegmented. This improvement is seemingly small but decidedly important for tweets where sentiment-related information is embedded in multi-word hashtags and sentiment prediction would be incorrect based only on the text (see Table TABREF60 for examples). In fact, 2,605 out of the 3,384 tweets have multi-word hashtags that contain words in the Twitter-based sentiment lexicon BIBREF44 and 125 tweets contain sentiment words only in the hashtags but not in the rest of the tweet. On the entire test set of 12,284 tweets, the increase in the average recall is 0.5%.\nOther Related Work\nAutomatic hashtag segmentation can improve the performance of many applications besides sentiment analysis, such as text classification BIBREF13 , named entity linking BIBREF10 and modeling user interests for recommendations BIBREF45 . It can also help in collecting data of higher volume and quality by providing a more nuanced interpretation of its content, as shown for emotion analysis BIBREF46 , sarcasm and irony detection BIBREF11 , BIBREF47 . Better semantic analysis of hashtags can also potentially be applied to hashtag annotation BIBREF48 , to improve distant supervision labels in training classifiers for tasks such as sarcasm BIBREF5 , sentiment BIBREF4 , emotions BIBREF3 ; and, more generally, as labels for pre-training representations of words BIBREF49 , sentences BIBREF50 , and images BIBREF51 .\nConclusion\nWe proposed a new pairwise neural ranking model for hashtag segmention and showed significant performance improvements over the state-of-the-art. We also constructed a larger and more curated dataset for analyzing and benchmarking hashtag segmentation methods. We demonstrated that hashtag segmentation helps with downstream tasks such as sentiment analysis. Although we focused on English hashtags, our pairwise ranking approach is language-independent and we intend to extend our toolkit to languages other than English as future work.\nAcknowledgments\nWe thank Ohio Supercomputer Center BIBREF52 for computing resources and the NVIDIA for providing GPU hardware. We thank Alan Ritter, Quanze Chen, Wang Ling, Pravar Mahajan, and Dushyanta Dhyani for valuable discussions. We also thank the annotators: Sarah Flanagan, Kaushik Mani, and Aswathnarayan Radhakrishnan. This material is based in part on research sponsored by the NSF under grants IIS-1822754 and IIS-1755898, DARPA through the ARO under agreement number W911NF-17-C-0095, through a Figure-Eight (CrowdFlower) AI for Everyone Award and a Criteo Faculty Research Award to Wei Xu. The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of the U.S. Government.\nWord-shape rules\nOur model uses the following word shape rules as boolean features. If the candidate segmentation INLINEFORM0 and its corresponding hashtag INLINEFORM1 satisfies a word shape rule, then the boolean feature is set to True.",
  "char_len": 24961,
  "document_hash": "0c1be0a4bce3ab32e3fc2c6ff239397d82513bfaa0b4f68d2e123765ac907401",
  "questions": [
    {
      "question_id": "q00",
      "qtype": "qasper",
      "question": "How is the dataset of hashtags sourced?"
    }
  ]
}