{
  "document_id": "lb_qasper_1b96562949f4722fed7766bb0be032ca10cb98c89192068f",
  "family": "public_qasper",
  "split": "test",
  "text": "Introduction\nIn the field of natural language processing (NLP), the most prevalent neural approach to obtaining sentence representations is to use recurrent neural networks (RNNs), where words in a sentence are processed in a sequential and recurrent manner. Along with their intuitive design, RNNs have shown outstanding performance across various NLP tasks e.g. language modeling BIBREF0 , BIBREF1 , machine translation BIBREF2 , BIBREF3 , BIBREF4 , text classification BIBREF5 , BIBREF6 , and parsing BIBREF7 , BIBREF8 .\nAmong several variants of the original RNN BIBREF9 , gated recurrent architectures such as long short-term memory (LSTM) BIBREF10 and gated recurrent unit (GRU) BIBREF2 have been accepted as de-facto standard choices for RNNs due to their capability of addressing the vanishing and exploding gradient problem and considering long-term dependencies. Gated RNNs achieve these properties by introducing additional gating units that learn to control the amount of information to be transferred or forgotten BIBREF11 , and are proven to work well without relying on complex optimization algorithms or careful initialization BIBREF12 .\nMeanwhile, the common practice for further enhancing the expressiveness of RNNs is to stack multiple RNN layers, each of which has distinct parameter sets (stacked RNN) BIBREF13 , BIBREF14 . In stacked RNNs, the hidden states of a layer are fed as input to the subsequent layer, and they are shown to work well due to increased depth BIBREF15 or their ability to capture hierarchical time series BIBREF16 which are inherent to the nature of the problem being modeled.\nHowever this setting of stacking RNNs might hinder the possibility of more sophisticated recurrence-based structures since the information from lower layers is simply treated as input to the next layer, rather than as another class of state that participates in core RNN computations. Especially for gated RNNs such as LSTMs and GRUs, this means that layer-to-layer connections cannot fully benefit from the carefully constructed gating mechanism used in temporal transitions. Some recent work on stacking RNNs suggests alternative methods that encourage direct and effective interaction between RNN layers by adding residual connections BIBREF17 , BIBREF18 , by shortcut connections BIBREF18 , BIBREF19 , or by using cell states of LSTMs BIBREF20 , BIBREF21 .\nIn this paper, we propose a method of constructing multi-layer LSTMs where cell states are used in controlling the vertical information flow. This system utilizes states from the left and the lower context equally in computation of the new state, thus the information from lower layers is elaborately filtered and reflected through a soft gating mechanism. Our method is easy-to-implement, effective, and can replace conventional stacked LSTMs without much modification of the overall architecture.\nWe call the proposed architecture Cell-aware Stacked LSTM, or CAS-LSTM, and evaluate our method on multiple benchmark datasets: SNLI BIBREF22 , MultiNLI BIBREF23 , Quora Question Pairs BIBREF24 , and SST BIBREF25 . From experiments we show that the CAS-LSTMs consistently outperform typical stacked LSTMs, opening the possibility of performance improvement of architectures that use stacked LSTMs.\nOur contribution is summarized as follows.\nThis paper is organized as follows. We give a detailed description about the proposed method in § SECREF2 . Experimental results are given in § SECREF3 . We study prior work related to our objective in § SECREF4 and conclude in § SECREF5 .\nModel Description\nIn this section, we give a detailed formulation of the architectures used in experiments.\nNotation\nThroughout this paper, we denote matrices as boldface capital letters ( INLINEFORM0 ), vectors as boldface lowercase letters ( INLINEFORM1 ), and scalars as normal italic letters ( INLINEFORM2 ). For LSTM states, we denote a hidden state as INLINEFORM3 and a cell state as INLINEFORM4 . Also, a layer index of INLINEFORM5 or INLINEFORM6 is denoted by superscript and a time index is denoted by a subscript, i.e. INLINEFORM7 indicates the hidden state at time INLINEFORM8 and layer INLINEFORM9 . INLINEFORM10 means the element-wise multiplication between two vectors. We write INLINEFORM11 -th component of vector INLINEFORM12 as INLINEFORM13 . All vectors are assumed to be column vectors.\nStacked LSTMs\nWhile there exist various versions of LSTM formulation, in this work we use the following, one of the most common versions: DISPLAYFORM0 DISPLAYFORM1\nwhere INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , INLINEFORM3 , INLINEFORM4 are trainable parameters. INLINEFORM5 and INLINEFORM6 are the sigmoid activation and the hyperbolic tangent activation function respectively. Also we assume that INLINEFORM7 where INLINEFORM8 is the INLINEFORM9 -th input to the network.\nThe input gate INLINEFORM0 and the forget gate INLINEFORM1 control the amount of information transmitted from INLINEFORM2 and INLINEFORM3 , the candidate cell state and the previous cell state, to the new cell state INLINEFORM4 . Similarly the output gate INLINEFORM5 soft-selects which portion of the cell state INLINEFORM6 is to be used in the final hidden state.\nWe can clearly see that cell states ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ) play a crucial role in forming horizontal recurrence. However the current formulation does not consider INLINEFORM3 , the cell state from INLINEFORM4 -th layer, in computation and thus the lower context is reflected only through the rudimentary way, hindering the possibility of controlling vertical information flow.\nCell-aware Stacked LSTMs\nNow we extend the stacked LSTM formulation defined above to address the problem noted in the previous subsection. To enhance the interaction between layers in a way similar to how LSTMs keep and forget the information from the previous time step, we introduce the additional forget gate INLINEFORM0 that determines whether to accept or ignore the signals coming from the previous layer. Therefore the proposed Cell-aware Stacked LSTM is formulated as follows: DISPLAYFORM0 DISPLAYFORM1\nwhere INLINEFORM0 and INLINEFORM1 . INLINEFORM2 can either be a vector of constants or parameters. When INLINEFORM3 , the equations defined in the previous subsection are used. Therefore, it can be said that each non-bottom layer of CAS-LSTM accepts two sets of hidden and cell states—one from the left context and the other from the below context. The left and the below context participate in computation with the equivalent procedure so that the information from lower layers can be efficiently propagated. Fig. FIGREF1 compares CAS-LSTM to the conventional stacked LSTM architecture, and Fig. FIGREF8 depicts the computation flow of the CAS-LSTM.\nWe argue that considering INLINEFORM0 in computation is beneficial for the following reasons. First, INLINEFORM1 contains additional information compared to INLINEFORM2 since it is not filtered by INLINEFORM3 . Thus a model that directly uses INLINEFORM4 does not rely solely on INLINEFORM5 for extracting information, due to the fact that it has access to the raw information INLINEFORM6 , as in temporal connections. In other words, INLINEFORM7 no longer has to take all responsibility for selecting useful features for both horizontal and vertical transitions, and the burden of selecting information is shared with INLINEFORM8 .\nAnother advantage of using the INLINEFORM0 lies in the fact that it directly connects INLINEFORM1 and INLINEFORM2 . This direct connection helps and stabilizes training, since the terminal error signals can be easily backpropagated to model parameters. Fig. FIGREF23 illustrates paths between the two cell states.\nWe find experimentally that there is little difference between letting INLINEFORM0 be constant and letting it be trainable parameters, thus we set INLINEFORM1 in all experiments. We also experimented with the architecture without INLINEFORM2 i.e. two cell states are combined by unweighted summation similar to multidimensional RNNs BIBREF27 , and found that it leads to performance degradation and unstable convergence, likely due to mismatch in the range of cell state values between layers ( INLINEFORM3 for the first layer and INLINEFORM4 for the others). Experimental results on various INLINEFORM5 are presented in § SECREF3 .\nThe idea of having multiple states is also related to tree-structured RNNs BIBREF29 , BIBREF30 . Among them, tree-structured LSTMs (Tree-LSTMs) BIBREF31 , BIBREF32 , BIBREF33 are similar to ours in that they use both hidden and cell states from children nodes. In Tree-LSTMs, states for all children nodes are regarded as input, and they participate in the computation equally through weight-shared (in Child-Sum Tree-LSTMs) or weight-unshared (in INLINEFORM0 -ary Tree-LSTMs) projection. From this perspective, each CAS-LSTM layer (where INLINEFORM1 ) can be seen as a binary Tree-LSTM where the structures it operates on are fixed to right-branching trees. The use of cell state in computation could be one reason that Tree-LSTMs perform better than sequential LSTMs even when trivial trees (strictly left- or right-branching) are given BIBREF34 .\nMultidimensional RNNs (MDRNN) are an extension of 1D sequential RNNs that can accept multidimensional input e.g. images, and have been successfully applied to image segmentation BIBREF26 and handwriting recognition BIBREF27 . Notably multidimensional LSTMs (MDLSTM) BIBREF27 have an analogous formulation to ours except the INLINEFORM0 term and the fact that we use distinct weights per column (or `layer' in our case). From this view, CAS-LSTM can be seen as a certain kind of MDLSTM that accepts a 2D input INLINEFORM1 . Grid LSTMs BIBREF21 also take INLINEFORM2 inputs but emit INLINEFORM3 outputs, which is different from our case where a single set of hidden and cell states is produced.\nSentence Encoders\nThe sentence encoder network we use in our experiments takes INLINEFORM0 words (assumed to be one-hot vectors) as input. The words are projected to corresponding word representations: INLINEFORM1 where INLINEFORM2 . Then INLINEFORM3 is fed to a INLINEFORM4 -layer CAS-LSTM model, resulting in the representations INLINEFORM5 . The sentence representation, INLINEFORM6 , is computed by max-pooling INLINEFORM7 over time as in the work of BIBREF35 . Similar to their results, from preliminary experiments we found that the max-pooling performs consistently better than mean- and last-pooling.\nTo make models more expressive, a bidirectional CAS-LSTM network may also be used. In the bidirectional case, the forward representations INLINEFORM0 and the backward representations INLINEFORM1 are concatenated and max-pooled to yield the sentence representation INLINEFORM2 . We call this bidirectional architecture Bi-CAS-LSTM in experiments.\nTop-layer Classifiers\nFor the natural language inference experiments, we use the following heuristic function proposed by BIBREF36 in feature extraction: DISPLAYFORM0\nwhere INLINEFORM0 means vector concatenation, and INLINEFORM1 and INLINEFORM2 are applied element-wise.\nAnd we use the following function in paraphrase identification experiments: DISPLAYFORM0\nas in the work of BIBREF37 .\nFor sentiment classification, we use the sentence representation itself. DISPLAYFORM0\nWe feed the feature extracted from INLINEFORM0 as input to the MLP classifier with ReLU activation followed by the fully-connected softmax layer to predict the label distribution: DISPLAYFORM0\nwhere INLINEFORM0 , INLINEFORM1 is the number of label classes, and INLINEFORM2 the dimension of the MLP output,\nExperiments\nWe evaluate our method on natural language inference (NLI), paraphrase identification (PI), and sentiment classification. We also conduct analysis on gate values and experiments on model variants. For detailed experimental settings, we refer readers to the supplemental material.\nFor the NLI and PI tasks, there exists recent work specializing in sentence pair classification. However in this work we confine our model to the architecture that encodes each sentence using a shared encoder without any inter-sentence interaction, in order to focus on the effectiveness of the models in extracting semantics. But note that the applicability of CAS-LSTM is not limited to sentence encoding based approaches.\nNatural Language Inference\nFor the evaluation of performance of the proposed method on the NLI task, SNLI BIBREF22 and MultiNLI BIBREF23 datasets are used. The objective of both datasets is to predict the relationship between a premise and a hypothesis sentence: entailment, contradiction, and neutral. SNLI and MultiNLI datasets are composed of about 570k and 430k premise-hypothesis pairs respectively.\nGloVe pretrained word embeddings BIBREF49 are used and remain fixed during training. The dimension of encoder states ( INLINEFORM0 ) is set to 300 and a 1024D MLP with one or two hidden layers is used. We apply dropout BIBREF50 to the word embeddings and the MLP layers. The features used as input to the MLP classifier are extracted following Eq. EQREF28 .\nTable TABREF32 and TABREF33 contain results of the models on SNLI and MultiNLI datasets. In SNLI, our best model achieves the new state-of-the-art accuracy of 87.0% with relatively fewer parameters. Similarly in MultiNLI, our models match the accuracy of state-of-the-art models in both in-domain (matched) and cross-domain (mismatched) test sets. Note that only the GloVe word vectors are used as word representations, as opposed to some models that introduce character-level features. It is also notable that our proposed architecture does not restrict the selection of pooling method; the performance could further be improved by replacing max-pooling with other advanced algorithms e.g. intra-sentence attention BIBREF39 and generalized pooling BIBREF19 .\nParaphrase Identification\nWe use Quora Question Pairs dataset BIBREF24 in evaluating the performance of our method on the PI task. The dataset consists of over 400k question pairs, and each pair is annotated with whether the two sentences are paraphrase of each other or not.\nSimilar to the NLI experiments, GloVe pretrained vectors, 300D encoders, and 1024D MLP are used. The number of CAS-LSTM layers is fixed to 2 in PI experiments. Two sentence vectors are aggregated using Eq. EQREF29 and fed as input to the MLP. The results on the Quora Question Pairs dataset are summarized in Table TABREF34 . Again we can see that our models outperform other models by large margin, achieving the new state of the art.\nSentiment Classification\nIn evaluating sentiment classification performance, the Stanford Sentiment Treebank (SST) BIBREF25 is used. It consists of about 12,000 binary-parsed sentences where constituents (phrases) of each parse tree are annotated with a sentiment label (very positive, positive, neutral, negative, very negative). Following the convention of prior work, all phrases and their labels are used in training but only the sentence-level data are used in evaluation.\nIn evaluation we consider two settings, namely SST-2 and SST-5, the two differing only in their level of granularity with regard to labels. In SST-2, data samples annotated with `neutral' are ignored from training and evaluation. The two positive labels (very positive, positive) are considered as the same label, and similarly for the two negative labels. As a result 98,794/872/1,821 data samples are used in training/validation/test, and the task is considered as a binary classification problem. In SST-5, data are used as-is and thus the task is a 5-class classification problem. All 318,582/1,101/2,210 data samples for training/validation/test are used in the SST-5 setting.\nWe use 300D GloVe vectors, 2-layer 150D or 300D encoders, and a 300D MLP classifier for the models, however unlike previous experiments we tune the word embeddings during training. The results on SST are listed in Table TABREF35 . Our models achieve the new state-of-the-art accuracy on SST-2 and competitive accuracy on SST-5, without utilizing parse tree information.\nForget Gate Analysis\nTo inspect the effect of the additional forget gate, we investigate how the values of vertical forget gates are distributed. We sample 1,000 random sentences from the development set of the SNLI dataset, and use the 3-layer CAS-LSTM model trained on the SNLI dataset to compute gate values.\nIf all values from a vertical forget gate INLINEFORM0 were to be 0, this would mean that the introduction of the additional forget gate is meaningless and the model would reduce to a plain stacked LSTM. On the contrary if all values were 1, meaning that the vertical forget gates were always open, it would be impossible to say that the information is modulated effectively.\nFig. FIGREF40 and FIGREF40 represent histograms of the vertical forget gate values from the second and the third layer. From the figures we can validate that the trained model does not fall into the degenerate case where vertical forget gates are ignored. Also the figures show that the values are right-skewed, which we conjecture to be a result of focusing more on a strong interaction between adjacent layers.\nTo further verify that the gate values are diverse enough within each time step, we compute the distribution of the range of values per time step, INLINEFORM0 , where INLINEFORM1 . We plot the histograms in Fig. FIGREF40 and FIGREF40 . From the figure we see that a vertical forget gate controls the amount of information flow effectively, making the decision of retaining or discarding signals.\nFinally, to investigate the argument presented in § SECREF2 that the additional forget gate helps the previous output gate with reducing the burden of extracting all needed information, we inspect the distribution of the values from INLINEFORM0 . This distribution indicates how differently the vertical forget gate and the previous output gate select information from INLINEFORM1 . From Fig. FIGREF40 and FIGREF40 we can see that the two gates make fairly different decisions, from which we demonstrate that the direct path between INLINEFORM2 and INLINEFORM3 enables a model to utilize signals overlooked by INLINEFORM4 .\nModel Variations\nIn this subsection, we see the influence of each component of a model on performance by removing or replacing its components. the SNLI dataset is used for experiments, and the best performing configuration is used as a baseline for modifications. We consider the following variants: (i) models that use plain stacked LSTMs, (ii) models with different INLINEFORM0 , (iii) models without INLINEFORM1 , and (iv) models that integrate lower contexts via peephole connections.\nVariant (iv) integrates lower contexts via the following equations: DISPLAYFORM0 DISPLAYFORM1\nwhere INLINEFORM0 represent peephole weights that take cell states into account. Among the above equations, those that use the lower cell state INLINEFORM1 are Eq. EQREF52 and EQREF55 . We can see that INLINEFORM2 affects the value of INLINEFORM3 only via peephole connections, which makes INLINEFORM4 independent of INLINEFORM5 .\nTable TABREF36 summarizes the results of model variants. We can again see that the use of cell states clearly improves sentence modeling performance (baseline vs. (i) and (iv) vs. (i)). Also from the results of baseline and (ii), we validate that the selection of INLINEFORM0 does not significantly affect performance but introducing INLINEFORM1 is beneficial (baseline vs. (iii)) possibly due to its effect on normalizing information from multiple sources, as mentioned in § SECREF2 . Finally, from the comparison between baseline and (iv), we show that the proposed way of combining the left and the lower contexts leads to better modeling of sentence representations than that of BIBREF20 in encoding sentences.\nConclusion\nIn this paper, we proposed a method of stacking multiple LSTM layers for modeling sentences, dubbed CAS-LSTM. It uses not only hidden states but also cell states from the previous layer, for the purpose of controlling the vertical information flow in a more elaborate way. We evaluated the proposed method on various benchmark tasks: natural language inference, paraphrase identification, and sentiment classification. Our models achieve the new state-of-the-art accuracy on SNLI and Quora Question Pairs datasets and obtain comparable results on MultiNLI and SST datasets. The proposed architecture can replace any stacked LSTM under one weak restriction—the size of states should be identical across all layers.\nFor future work we plan to apply the CAS-LSTM architecture beyond sentence modeling tasks. Various problems e.g. sequence labeling, sequence generation, and language modeling might benefit from sophisticated modulation on context integration. Aggregating diverse contexts from sequential data, e.g. those from forward and backward reading of text, could also be an intriguing research direction.\nAcknowledgments\nWe thank Dan Edmiston for the review of the manuscript.",
  "char_len": 21057,
  "document_hash": "abf8d069e619adf6f6b10447978eada81004b5fbfb8e4ff170496c4ac5f156ec",
  "questions": [
    {
      "question_id": "q00",
      "qtype": "qasper",
      "question": "Which models did they experiment with?"
    }
  ]
}