spb/wp30_uqo

1{2  "document_id": "lb_qasper_3598040772b4950e6241b50125f7ab7038a8149060e3c381",3  "family": "public_qasper",4  "split": "test",5  "text": "Introduction\nRecently, deep neural network has been widely employed in various recognition tasks. Increasing the depth of neural network is a effective way to improve the performance, and convolutional neural network (CNN) has benefited from it in visual recognition task BIBREF0 . Deeper long short-term memory (LSTM) recurrent neural networks (RNNs) are also applied in large vocabulary continuous speech recognition (LVCSR) task, because LSTM networks have shown better performance than Fully-connected feed-forward deep neural network BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 .\nTraining neural network becomes more challenge when it goes deep. A conceptual tool called linear classifier probe is introduced to better understand the dynamics inside a neural network BIBREF5 . The discriminating features of linear classifier is the hidden units of a intermediate layer. For deep neural networks, it is observed that deeper layer's accuracy is lower than that of shallower layers. Therefore, the tool shows the difficulty of deep neural model training visually.\nLayer-wise pre-training is a successful method to train very deep neural networks BIBREF6 . The convergence becomes harder with increasing the number of layers, even though the model is initialized with Xavier or its variants BIBREF7 , BIBREF8 . But the deeper network which is initialized with a shallower trained network could converge well.\nThe size of LVCSR training dataset goes larger and training with only one GPU becomes high time consumption inevitably. Therefore, parallel training with multi-GPUs is more suitable for LVCSR system. Mini-batch based stochastic gradient descent (SGD) is the most popular method in neural network training procedure. Asynchronous SGD is a successful effort for parallel training based on it BIBREF9 , BIBREF10 . It can many times speed up the training time without decreasing the accuracy. Besides, synchronous SGD is another effective effort, where the parameter server waits for every works to finish their computation and sent their local models to it, and then it sends updated model back to all workers BIBREF11 . Synchronous SGD converges well in parallel training with data parallelism, and is also easy to implement.\nIn order to further improve the performance of deep neural network with parallel training, several methods are proposed. Model averaging method achieves linear speedup, as the final model is averaged from all parameters of local models in different workers BIBREF12 , BIBREF13 , but the accuracy decreases compared with single GPU training. Moreover, blockwise model-updating filter (BMUF) provides another almost linear speedup approach with multi-GPUs on the basis of model averaging. It can achieve improvement or no-degradation of recognition performance compared with mini-batch SGD on single GPU BIBREF14 .\nMoving averaged (MA) approaches are also proposed for parallel training. It is demonstrated that the moving average of the parameters obtained by SGD performs as well as the parameters that minimize the empirical cost, and moving average parameters can be used as the estimator of them, if the size of training data is large enough BIBREF15 . One pass learning is then proposed, which is the combination of learning rate schedule and averaged SGD using moving average BIBREF16 . Exponential moving average (EMA) is proposed as a non-interference method BIBREF17 . EMA model is not broadcasted to workers to update their local models, and it is applied as the final model of entire training process. EMA method is utilized with model averaging and BMUF to further decrease the character error rate (CER). It is also easy to implement in existing parallel training systems.\nFrame stacking can also speed up the training time BIBREF18 . The super frame is stacked by several regular frames, and it contains the information of them. Thus, the network can see multiple frames at a time, as the super frame is new input. Frame stacking can also lead to faster decoding.\nFor streaming voice search service, it needs to display intermediate recognition results while users are still speaking. As a result, the system needs to fulfill high real-time requirement, and we prefer unidirectional LSTM network rather than bidirectional one. High real-time requirement means low real time factor (RTF), but the RTF of deep LSTM model is higher inevitably. The dilemma of recognition accuracy and real-time requirement is an obstacle to the employment of deep LSTM network. Deep model outperforms because it contains more knowledge, but it is also cumbersome. As a result, the knowledge of deep model can be distilled to a shallow model BIBREF19 . It provided a effective way to employ the deep model to the real-time system.\nIn this paper, we explore a entire deep LSTM RNN training framework, and employ it to real-time application. The deep learning systems benefit highly from a large quantity of labeled training data. Our first and basic speech recognition system is trained on 17000 hours of Shenma voice search dataset. It is a generic dataset sampled from diverse aspects of search queries. The requirement of speech recognition system also addressed by specific scenario, such as map and navigation task. The labeled dataset is too expensive, and training a new model with new large dataset from the beginning costs lots of time. Thus, it is natural to think of transferring the knowledge from basic model to new scenario's model. Transfer learning expends less data and less training time than full training. In this paper, we also introduce a novel transfer learning strategy with segmental Minimum Bayes-Risk (sMBR). As a result, transfer training with only 1000 hours data can match equivalent performance for full training with 7300 hours data.\nOur deep LSTM training framework for LVCSR is presented in Section 2. Section 3 describes how the very deep models does apply in real world applications, and how to transfer the model to another task. The framework is analyzed and discussed in Section 4, and followed by the conclusion in Section 5.\nLayer-wise Training with Soft Target and Hard Target\nGradient-based optimization of deep LSTM network with random initialization get stuck in poor solution easily. Xavier initialization can partially solve this problem BIBREF7 , so this method is the regular initialization method of all training procedure. However, it does not work well when it is utilized to initialize very deep model directly, because of vanishing or exploding gradients. Instead, layer-wise pre-training method is a effective way to train the weights of very deep architecture BIBREF6 , BIBREF20 . In layer-wise pre-training procedure, a one-layer LSTM model is firstly trained with normalized initialization. Sequentially, two-layers LSTM model's first layer is initialized by trained one-layer model, and its second layer is regularly initialized. In this way, a deep architecture is layer-by-layer trained, and it can converge well.\nIn conventional layer-wise pre-training, only parameters of shallower network are transfered to deeper one, and the learning targets are still the alignments generated by HMM-GMM system. The targets are vectors that only one state's probability is one, and the others' are zeros. They are known as hard targets, and they carry limited knowledge as only one state is active. In contrast, the knowledge of shallower network should be also transfered to deeper one. It is obtained by the softmax layer of existing model typically, so each state has a probability rather than only zero or one, and called as soft target. As a result, the deeper network which is student network learns the parameters and knowledge from shallower one which is called teacher network. When training the student network from the teacher network, the final alignment is the combination of hard target and soft target in our layer-wise training phase. The final alignment provides various knowledge which transfered from teacher network and extracted from true labels. If only soft target is learned, student network perform no better than teacher network, but it could outperform teacher network as it also learns true labels.\nThe deeper network spends less time to getting the same level of original network than the network trained from the beginning, as a period of low performance is skipped. Therefore, training with hard and soft target is a time saving method. For large training dataset, training with the whole dataset still spends too much time. A network firstly trained with only a small part of dataset could go deeper as well, and so the training time reducing rapidly. When the network is deep enough, it then trained on the entire dataset to get further improvement. There is no gap of accuracy between these two approaches, but latter one saves much time.\nDifferential Saturation Check\nThe objects of conventional saturation check are gradients and the cell activations BIBREF4 . Gradients are clipped to range [-5, 5], while the cell activations clipped to range [-50, 50]. Apart from them, the differentials of recurrent layers is also limited. If the differentials go beyond the range, corresponding back propagation is skipped, while if the gradients and cell activations go beyond the bound, values are set as the boundary values. The differentials which are too large or too small will lead to the gradients easily vanishing, and it demonstrates the failure of this propagation. As a result, the parameters are not updated, and next propagation .\nSequence Discriminative Training\nCross-entropy (CE) is widely used in speech recognition training system as a frame-wise discriminative training criterion. However, it is not well suited to speech recognition, because speech recognition training is a sequential learning problem. In contrast, sequence discriminative training criterion has shown to further improve performance of neural network first trained with cross-entropy BIBREF21 , BIBREF22 , BIBREF23 . We choose state-level minimum bayes risk (sMBR) BIBREF21 among a number of sequence discriminative criterion is proposed, such as maximum mutual information (MMI) BIBREF24 and minimum phone error (MPE) BIBREF25 . MPE and sMBR are designed to minimize the expected error of different granularity of labels, while CE aims to minimizes expected frame error, and MMI aims to minimizes expected sentence error. State-level information is focused on by sMBR.\na frame-level accurate model is firstly trained by CE loss function, and then sMBR loss function is utilized for further training to get sequence-level accuracy. Only a part of training dataset is needed in sMBR training phase on the basis of whole dataset CE training.\nParallel Training\nIt is demonstrated that training with larger dataset can improve recognition accuracy. However, larger dataset means more training samples and more model parameters. Therefore, parallel training with multiple GPUs is essential, and it makes use of data parallelism BIBREF9 . The entire training data is partitioned into several split without overlapping and they are distributed to different GPUs. Each GPU trains with one split of training dataset locally. All GPUs synchronize their local models with model average method after a mini-batch optimization BIBREF12 , BIBREF13 .\nModel average method achieves linear speedup in training phase, but the recognition accuracy decreases compared with single GPU training. Block-wise model updating filter (BMUF) is another successful effort in parallel training with linear speedup as well. It can achieve no-degradation of recognition accuracy with multi-GPUs BIBREF14 . In the model average method, aggregated model INLINEFORM0 is computed and broadcasted to GPUs. On the basis of it, BMUF proposed a novel model updating strategy: INLINEFORM1 INLINEFORM2\nWhere INLINEFORM0 denotes model update, and INLINEFORM1 is the global-model update. There are two parameters in BMUF, block momentum INLINEFORM2 , and block learning rate INLINEFORM3 . Then, the global model is updated as INLINEFORM4\nConsequently, INLINEFORM0 is broadcasted to all GPUs to initial their local models, instead of INLINEFORM1 in model average method.\nAveraged SGD is proposed to further accelerate the convergence speed of SGD. Averaged SGD leverages the moving average (MA) INLINEFORM0 as the estimator of INLINEFORM1 BIBREF15 : INLINEFORM2\nWhere INLINEFORM0 is computed by model averaging or BMUF. It is shown that INLINEFORM1 can well converge to INLINEFORM2 , with the large enough training dataset in single GPU training. It can be considered as a non-interference strategy that INLINEFORM3 does not participate the main optimization process, and only takes effect after the end of entire optimization. However, for the parallel training implementation, each INLINEFORM4 is computed by model averaging and BMUF with multiple models, and moving average model INLINEFORM5 does not well converge, compared with single GPU training.\nModel averaging based methods are employed in parallel training of large scale dataset, because of their faster convergence, and especially no-degradation implementation of BMUF. But combination of model averaged based methods and moving average does not match the expectation of further enhance performance and it is presented as INLINEFORM0\nThe weight of each INLINEFORM0 is equal in moving average method regardless the effect of temporal order. But INLINEFORM1 closer to the end of training achieve higher accuracy in the model averaging based approach, and thus it should be with more proportion in final INLINEFORM2 . As a result, exponential moving average(EMA) is appropriate, which the weight for each older parameters decrease exponentially, and never reaching zero. After moving average based methods, the EMA parameters are updated recursively as INLINEFORM3\nHere INLINEFORM0 represents the degree of weight decrease, and called exponential updating rate. EMA is also a non-interference training strategy that is implemented easily, as the updated model is not broadcasted. Therefore, there is no need to add extra learning rate updating approach, as it can be appended to existing training procedure directly.\nDeployment\nThere is a high real time requirement in real world application, especially in online voice search system. Shenma voice search is one of the most popular mobile search engines in China, and it is a streaming service that intermediate recognition results displayed while users are still speaking. Unidirectional LSTM network is applied, rather than bidirectional one, because it is well suited to real-time streaming speech recognition.\nDistillation\nIt is demonstrated that deep neural network architecture can achieve improvement in LVCSR. However, it also leads to much more computation and higher RTF, so that the recognition result can not be displayed in real time. It should be noted that deeper neural network contains more knowledge, but it is also cumbersome. the knowledge is key to improve the performance. If it can be transfered from cumbersome model to a small model, the recognition ability can also be transfered to the small model. Knowledge transferring to small model is called distillation BIBREF19 . The small model can perform as well as cumbersome model, after distilling. It provide a way to utilize high-performance but high RTF model in real time system. The class probability produced by the cumbersome model is regarded as soft target, and the generalization ability of cumbersome model is transfered to small model with it. Distillation is model's knowledge transferring approach, so there is no need to use the hard target, which is different with the layer-wise training method.\n9-layers unidirectional LSTM model achieves outstanding performance, but meanwhile it is too computationally expensive to allow deployment in real time recognition system. In order to ensure real-time of the system, the number of layers needs to be reduced. The shallower network can learn the knowledge of deeper network with distillation. It is found that RTF of 2-layers network is acceptable, so the knowledge is distilled from 9-layers well-trained model to 2-layers model. Table TABREF16 shows that distillation from 9-layers to 2-layers brings RTF decrease of relative 53%, while CER only increases 5%. The knowledge of deep network is almost transfered with distillation, Distillation brings promising RTF reduction, but only little knowledge of deep network is lost. Moreover, CER of 2-layers distilled LSTM decreases relative 14%, compared with 2-layers regular-trained LSTM.\nTransfer Learning with sMBR\nFor a certain specific scenario, the model trained with the data recorded from it has better adaptation than the model trained with generic scenario. But it spends too much time training a model from the beginning, if there is a well-trained model for generic scenarios. Moreover, labeling a large quantity of training data in new scenario is both costly and time consuming. If a model transfer trained with smaller dataset can obtained the similar recognition accuracy compared with the model directly trained with larger dataset, it is no doubt that transfer learning is more practical. Since specific scenario is a subset of generic scenario, some knowledge can be shared between them. Besides, generic scenario consists of various conditions, so its model has greater robustness. As a result, not only shared knowledge but also robustness can be transfered from the model of generic scenario to the model of specific one.\nAs the model well trained from generic scenario achieves good performance in frame level classification, sequence discriminative training is required to adapt new model to specific scenario additionally. Moreover, it does not need alignment from HMM-GMM system, and it also saves amount of time to prepare alignment.\nTraining Data\nA large quantity of labeled data is needed for training a more accurate acoustic model. We collect the 17000 hours labeled data from Shenma voice search, which is one of the most popular mobile search engines in China. The dataset is created from anonymous online users' search queries in Mandarin, and all audio file's sampling rate is 16kHz, recorded by mobile phones. This dataset consists of many different conditions, such as diverse noise even low signal-to-noise, babble, dialects, accents, hesitation and so on.\nIn the Amap, which is one of the most popular web mapping and navigation services in China, users can search locations and navigate to locations they want though voice search. To present the performance of transfer learning with sequence discriminative training, the model trained from Shenma voice search which is greneric scenario transfer its knowledge to the model of Amap voice search. 7300 hours labeled data is collected in the similar way of Shenma voice search data collection.\nTwo dataset is divided into training set, validation set and test set separately, and the quantity of them is shown in Table TABREF10 . The three sets are split according to speakers, in order to avoid utterances of same speaker appearing in three sets simultaneously. The test sets of Shenma and Amap voice search are called Shenma Test and Amap Test.\nExperimental setup\nLSTM RNNs outperform conventional RNNs for speech recognition system, especially deep LSTM RNNs, because of its long-range dependencies more accurately for temporal sequence conditions BIBREF26 , BIBREF23 . Shenma and Amap voice search is a streaming service that intermediate recognition results displayed while users are still speaking. So as for online recognition in real time, we prefer unidirectional LSTM model rather than bidirectional one. Thus, the training system is unidirectional LSTM-based.\nA 26-dimensional filter bank and 2-dimensional pitch feature is extracted for each frame, and is concatenated with first and second order difference as the final input of the network. The super frame are stacked by 3 frames without overlapping. The architecture we trained consists of two LSTM layers with sigmoid activation function, followed by a full-connection layer. The out layer is a softmax layer with 11088 hidden markov model (HMM) tied-states as output classes, the loss function is cross-entropy (CE). The performance metric of the system in Mandarin is reported with character error rate (CER). The alignment of frame-level ground truth is obtained by GMM-HMM system. Mini-batched SGD is utilized with momentum trick and the network is trained for a total of 4 epochs. The block learning rate and block momentum of BMUF are set as 1 and 0.9. 5-gram language model is leveraged in decoder, and the vocabulary size is as large as 760000. Differentials of recurrent layers is limited to range [-10000,10000], while gradients are clipped to range [-5, 5] and cell activations clipped to range [-50, 50]. After training with CE loss, sMBR loss is employed to further improve the performance.\nIt has shown that BMUF outperforms traditional model averaging method, and it is utilized at the synchronization phase. After synchronizing with BMUF, EMA method further updates the model in non-interference way. The training system is deployed on the MPI-based HPC cluster where 8 GPUs. Each GPU processes non-overlap subset split from the entire large scale dataset in parallel.\nLocal models from distributed workers synchronize with each other in decentralized way. In the traditional model averaging and BMUF method, a parameter server waits for all workers to send their local models, aggregate them, and send the updated model to all workers. Computing resource of workers is wasted until aggregation of the parameter server done. Decentralized method makes full use of computing resource, and we employ the MPI-based Mesh AllReduce method. It is mesh topology as shown in Figure FIGREF12 . There is no centralized parameter server, and peer to peer communication is used to transmit local models between workers. Local model INLINEFORM0 of INLINEFORM1 -th worker in INLINEFORM2 workers cluster is split to INLINEFORM3 pieces INLINEFORM4 , and send to corresponding worker. In the aggregation phase, INLINEFORM5 -th worker computed INLINEFORM6 splits of model INLINEFORM7 and send updated model INLINEFORM8 back to workers. As a result, all workers participate in aggregation and no computing resource is dissipated. It is significant to promote training efficiency, when the size of neural network model is too large. The EMA model is also updated additionally, but not broadcasting it.\nResults\nIn order to evaluate our system, several sets of experiments are performed. The Shenma test set including about 9000 samples and Amap test set including about 7000 samples contain various real world conditions. It simulates the majority of user scenarios, and can well evaluates the performance of a trained model. Firstly, we show the results of models trained with EMA method. Secondly, for real world applications, very deep LSTM is distilled to a shallow one, so as for lower RTF. The model of Amap is also needed to train for map and navigation scenarios. The performance of transfer learning from Shenma voice search to Amap voice search is also presented.\nLayer-wise Training\nIn layer-wise training, the deeper model learns both parameters and knowledge from the shallower model. The deeper model is initialized by the shallower one, and its alignment is the combination of hard target and soft target of shallower one. Two targets have the same weights in our framework. The teacher model is trained with CE. For each layer-wise trained CE model, corresponding sMBR model is also trained, as sMBR could achieve additional improvement. In our framework, 1000 hours data is randomly selected from the total dataset for sMBR training. There is no obvious performance enhancement when the size of sMBR training dataset increases.\nFor very deep unidirectional LSTM initialized with Xavier initialization algorithm, 6-layers model converges well, but there is no further improvement with increasing the number of layers. Therefore, the first 6 layers of 7-layers model is initialized by 6-layers model, and soft target is provided by 6-layers model. Consequently, deeper LSTM is also trained in the same way. It should be noticed that the teacher model of 9-layers model is the 8-layers model trained by sMBR, while the other teacher model is CE model. As shown in Table TABREF15 , the layer-wise trained models always performs better than the models with Xavier initialization, as the model is deep. Therefore, for the last layer training, we choose 8-layers sMBR model as the teacher model instead of CE model. A comparison between 6-layers and 9-layers sMBR models shows that 3 additional layers of layer-wise training brings relative 12.6% decreasing of CER. It is also significant that the averaged CER of sMBR models with different layers decreases absolute 0.73% approximately compared with CE models, so the improvement of sequence discriminative learning is promising.\nTransfer Learning\n2-layers distilled model of Shenma voice search has shown a impressive performance on Shenma Test, and we call it Shenma model. It is trained for generic search scenario, but it has less adaptation for specific scenario like Amap voice search. Training with very large dataset using CE loss is regarded as improvement of frame level recognition accuracy, and sMBR with less dataset further improves accuracy as sequence discriminative training. If robust model of generic scenario is trained, there is no need to train a model with very large dataset, and sequence discriminative training with less dataset is enough. Therefore, on the basis of Shenma model, it is sufficient to train a new Amap model with small dataset using sMBR. As shown in Table TABREF18 , Shenma model presents the worst performance among three methods, since it does not trained for Amap scenario. 2-layers Shenma model further trained with sMBR achieves about 8.1% relative reduction, compared with 2-layers regular-trained Amap model. Both training sMBR datasets contain the same 1000 hours data. As a result, with the Shenma model, only about 14% data usage achieves lower CER, and it leads to great time and cost saving with less labeled data. Besides, transfer learning with sMBR does not use the alignment from the HMM-GMM system, so it also saves huge amount of time.\nConclusion\nWe have presented a whole deep unidirectional LSTM parallel training system for LVCSR. The recognition performance improves when the network goes deep. Distillation makes it possible that deep LSTM model transfer its knowledge to shallow model with little loss. The model could be distilled to 2-layers model with very low RTF, so that it can display the immediate recognition results. As a result, its CER decrease relatively 14%, compared with the 2-layers regular trained model. In addition, transfer learning with sMBR is also proposed. If a great model has well trained from generic scenario, only 14% of the size of training dataset is needed to train a more accuracy acoustic model for specific scenario. Our future work includes 1) finding more effective methods to reduce the CER by increasing the number of layers; 2) applying this training framework to Connectionist Temporal Classification (CTC) and attention-based neural networks.",6  "char_len": 27395,7  "document_hash": "df748158dc995568c1f3ae46a3f5d041140346ee8b195cde83dceda049f126bd",8  "questions": [9    {10      "question_id": "q00",11      "qtype": "qasper",12      "question": "which lstm models did they compare with?"13    }14  ]15}