{
  "document_id": "lb_qasper_92da01e7242f30f5266e431b4269fee1b0ca5fcc23aee095",
  "family": "public_qasper",
  "split": "test",
  "text": "Introduction\nSuppose a user wants to write a sentence “I will be 10 minutes late.” Ideally, she would type just a few keywords such as “10 minutes late” and an autocomplete system would be able to infer the intended sentence (Figure FIGREF1). Existing left-to-right autocomplete systems BIBREF0, BIBREF1 can often be inefficient, as the prefix of a sentence (e.g. “I will be”) fails to capture the core meaning of the sentence. Besides the practical goal of building a better autocomplete system, we are interested in exploring the tradeoffs inherent to such communication schemes between the efficiency of typing keywords, accuracy of reconstruction, and interpretability of keywords.\nOne approach to learn such schemes is to collect a supervised dataset of keywords-sentence pairs as a training set, but (i) it would be expensive to collect such data from users, and (ii) a static dataset would not capture a real user's natural predilection to adapt to the system BIBREF2. Another approach is to avoid supervision and jointly learn a user-system communication scheme to directly optimize the combination of efficiency and accuracy. However, learning in this way can lead to communication schemes that are uninterpretable to humans BIBREF3, BIBREF4 (see Appendix for additional related work).\nIn this work, we propose a simple, unsupervised approach to an autocomplete system that is efficient, accurate, and interpretable. For interpretability, we restrict keywords to be subsequences of their source sentences based on the intuition that humans can infer most of the original meaning from a few keywords. We then apply multi-objective optimization approaches to directly control and achieve desirable tradeoffs between efficiency and accuracy.\nWe observe that naively optimizing a linear combination of efficiency and accuracy terms is unstable and leads to suboptimal schemes. Thus, we propose a new objective which optimizes for communication efficiency under an accuracy constraint. We show this new objective is more stable and efficient than the linear objective at all accuracy levels.\nAs a proof-of-concept, we build an autocomplete system within this framework which allows a user to write sentences by specifying keywords. We empirically show that our framework produces communication schemes that are 52.16% more accurate than rule-based baselines when specifying 77.37% of sentences, and 11.73% more accurate than a naive, weighted optimization approach when specifying 53.38% of sentences. Finally, we demonstrate that humans can easily adapt to the keyword-based autocomplete system and save nearly 50% of time compared to typing a full sentence in our user study.\nApproach\nConsider a communication game in which the goal is for a user to communicate a target sequence $x= (x_1, ..., x_m)$ to a system by passing a sequence of keywords $z= (z_1, ..., z_n)$. The user generates keywords $z$ using an encoding strategy $q_{\\alpha }(z\\mid x)$, and the system attempts to guess the target sequence $x$ via a decoding strategy $p_{\\beta }(x\\mid z)$.\nA good communication scheme $(q_{\\alpha }, p_{\\beta })$ should be both efficient and accurate. Specifically, we prefer schemes that use fewer keywords (cost), and the target sentence $x$ to be reconstructed with high probability (loss) where\nBased on our assumption that humans have an intuitive sense of retaining important keywords, we restrict the set of schemes to be a (potentially noncontiguous) subsequence of the target sentence. Our hypothesis is that such subsequence schemes naturally ensure interpretability, as efficient human and machine communication schemes are both likely to involve keeping important content words.\nApproach ::: Modeling with autoencoders.\nTo learn communication schemes without supervision, we model the cooperative communication between a user and system through an encoder-decoder framework. Concretely, we model the user's encoding strategy $q_{\\alpha }(z\\mid x)$ with an encoder which encodes the target sentence $x$ into the keywords $z$ by keeping a subset of the tokens. This stochastic encoder $q_{\\alpha }(z\\mid x)$ is defined by a model which returns the probability of each token retained in the final subsequence $z$. Then, we sample from Bernoulli distributions according to these probabilities to either keep or drop the tokens independently (see Appendix for an example).\nWe model the autocomplete system's decoding strategy $p_{\\beta }(x\\mid z)$ as a probabilistic model which conditions on the keywords $z$ and returns a distribution over predictions $x$. We use a standard sequence-to-sequence model with attention and copying for the decoder, but any model architecture can be used (see Appendix for details).\nApproach ::: Multi-objective optimization.\nOur goal now is to learn encoder-decoder pairs which optimally balance the communication cost and reconstruction loss. The simplest approach to balancing efficiency and accuracy is to weight $\\mathrm {cost}(x, \\alpha )$ and $\\mathrm {loss}(x, \\alpha , \\beta )$ linearly using a weight $\\lambda $ as follows,\nwhere the expectation is taken over the population distribution of source sentences $x$, which is omitted to simplify notation. However, we observe that naively weighting and searching over $\\lambda $ is suboptimal and highly unstable—even slight changes to the weighting results in degenerate schemes which keep all or none of its tokens. This instability motivates us to develop a new stable objective.\nOur main technical contribution is to draw inspiration from the multi-objective optimization literature and view the tradeoff as a sequence of constrained optimization problems, where we minimize the expected cost subject to varying expected reconstruction error constraints $\\epsilon $,\nThis greatly improves the stability of the training procedure. We empirically observe that the model initially keeps most of the tokens to meet the constraints, and slowly learns to drop uninformative words from the keywords to minimize the cost. Furthermore, $\\epsilon $ in Eq (DISPLAY_FORM6) allows us to directly control the maximum reconstruction error of resulting schemes, whereas $\\lambda $ in Eq (DISPLAY_FORM5) is not directly related to any of our desiderata.\nTo optimize the constrained objective, we consider the Lagrangian of Eq (DISPLAY_FORM6),\nMuch like the objective in Eq (DISPLAY_FORM5) we can compute unbiased gradients by replacing the expectations with their averages over random minibatches. Although gradient descent guarantees convergence on Eq (DISPLAY_FORM7) only when the objective is convex, we find that not only is the optimization stable, the resulting solution achieves better performance than the weighting approach in Eq (DISPLAY_FORM5).\nApproach ::: Optimization.\nOptimization with respect to $q_{\\alpha }(z\\mid x)$ is challenging as $z$ is discrete, and thus, we cannot differentiate $\\alpha $ through $z$ via the chain rule. Because of this, we use the stochastic REINFORCE estimate BIBREF5 as follows:\nWe perform joint updates on $(\\alpha , \\beta , \\lambda )$, where $\\beta $ and $\\lambda $ are updated via standard gradient computations, while $\\alpha $ uses an unbiased, stochastic gradient estimate where we approximate the expectation in Eq (DISPLAY_FORM9). We use a single sample from $q_{\\alpha }(z\\mid x)$ and moving-average of rewards as a baseline to reduce variance.\nExperiments\nWe evaluate our approach by training an autocomplete system on 500K randomly sampled sentences from Yelp reviews BIBREF6 (see Appendix for details). We quantify the efficiency of a communication scheme $(q_{\\alpha },p_{\\beta })$ by the retention rate of tokens, which is measured as the fraction of tokens that are kept in the keywords. The accuracy of a scheme is measured as the fraction of sentences generated by greedily decoding the model that exactly matches the target sentence.\nExperiments ::: Effectiveness of constrained objective.\nWe first show that the linear objective in Eq (DISPLAY_FORM5) is suboptimal compared to the constrained objective in Eq (DISPLAY_FORM6). Figure FIGREF10 compares the achievable accuracy and efficiency tradeoffs for the two objectives, which shows that the constrained objective results in more efficient schemes than the linear objective at every accuracy level (e.g. 11.73% more accurate at a 53.38% retention rate).\nWe also observe that the linear objective is highly unstable as a function of the tradeoff parameter $\\lambda $ and requires careful tuning. Even slight changes to $\\lambda $ results in degenerate schemes that keep all or none of the tokens (e.g. $\\lambda \\le 4.2$ and $\\lambda \\ge 4.4$). On the other hand, the constrained objective is substantially more stable as a function of $\\epsilon $ (e.g. points for $\\epsilon $ are more evenly spaced than $\\lambda $).\nExperiments ::: Efficiency-accuracy tradeoff.\nWe quantify the efficiency-accuracy tradeoff compared to two rule-based baselines: Unif and Stopword. The Unif encoder randomly keeps tokens to generate keywords with the probability $\\delta $. The Stopword encoder keeps all tokens but drops stop words (e.g. `the', `a', `or') all the time ($\\delta =0$) or half of the time ($\\delta =0.5$). The corresponding decoders for these encoders are optimized using gradient descent to minimize the reconstruction error (i.e. $\\mathrm {loss}(x, \\alpha , \\beta )$).\nFigure FIGREF10 shows that two baselines achieve similar tradeoff curves, while the constrained model achieves a substantial 52.16% improvement in accuracy at a 77.37% retention rate compared to Unif, thereby showing the benefits of jointly training the encoder and decoder.\nExperiments ::: Robustness and analysis.\nWe provide additional experimental results on the robustness of learned communication schemes as well as in-depth analysis on the correlation between the retention rates of tokens and their properties, which we defer to Appendix and for space.\nExperiments ::: User study.\nWe recruited 100 crowdworkers on Amazon Mechanical Turk (AMT) and measured completion times and accuracies for typing randomly sampled sentences from the Yelp corpus. Each user was shown alternating autocomplete and writing tasks across 50 sentences (see Appendix for user interface). For the autocomplete task, we gave users a target sentence and asked them to type a set of keywords into the system. The users were shown the top three suggestions from the autocomplete system, and were asked to mark whether each of these three suggestions was semantically equivalent to the target sentence. For the writing task, we gave users a target sentence and asked them to either type the sentence verbatim or a sentence that preserves the meaning of the target sentence.\nTable TABREF13 shows two examples of the autocomplete task and actual user-provided keywords. Each column contains a set of keywords and its corresponding top three suggestions generated by the autocomplete system with beam search. We observe that the system is likely to propose generic sentences for under-specified keywords (left column) and almost the same sentences for over-specified keywords (right column). For properly specified keywords (middle column), the system completes sentences accordingly by adding a verb, adverb, adjective, preposition, capitalization, and punctuation.\nOverall, the autocomplete system achieved high accuracy in reconstructing the keywords. Users marked the top suggestion from the autocomplete system to be semantically equivalent to the target $80.6$% of the time, and one of the top 3 was semantically equivalent $90.11$% of the time. The model also achieved a high exact match accuracy of 18.39%. Furthermore, the system was efficient, as users spent $3.86$ seconds typing keywords compared to $5.76$ seconds for full sentences on average. The variance of the typing time was $0.08$ second for keywords and $0.12$ second for full sentences, indicating that choosing and typing keywords for the system did not incur much overhead.\nExperiments ::: Acknowledgments\nWe thank the reviewers and Yunseok Jang for their insightful comments. This work was supported by NSF CAREER Award IIS-1552635 and an Intuit Research Award.\nExperiments ::: Reproducibility\nAll code, data and experiments are available on CodaLab at https://bit.ly/353fbyn.",
  "char_len": 12299,
  "document_hash": "9d5b2acc8346d6e1f64cc3c76c1c46b2edad499c17909fa18f8c1a4c20b844d7",
  "questions": [
    {
      "question_id": "q00",
      "qtype": "qasper",
      "question": "How are models evaluated in this human-machine communication game?"
    }
  ]
}