1{2 "document_id": "lb_qasper_d999910bed46747299b4c315cc345dc978d979252f65d39e",3 "family": "public_qasper",4 "split": "test",5 "text": "Introduction\nDeep neural networks (DNNs), in particular convolutional and recurrent neural networks, with huge architectures have been proven successful in wide range of tasks including audio processing such as speech to text [1 - 4], emotion recognition [5 - 8], speech/non-speech (e.g., of non-speech include noise, music, etc.,) classification [9 - 12], etc.\nTraining these deep architectures require large amount of annotated data, as a result, they cannot be used in low data resource scenarios which is common in speech-based applications [13 - 15]. Apart from collecting large data corpus, annotating the data is also very difficult, and requires manual supervision and efforts. Especially, annotation of speech for tasks like emotion recognition also suffer from lack of agreement among the annotators [16]. Hence, there is a need to build reliable systems that can work in low resource scenario.\nIn this work, we propose a novel approach to address the task of classification in low data resource scenarios. Our approach involves simultaneously considering more than one sample (in this work, two samples are considered) to train the classifier. We call this approach as simultaneous two sample learning (s2sL). The proposed approach is also applicable to low resource data suffering with data imbalance. The contributions of this paper are:\nProposed approach\nThe s2sL approach proposed to address low data resource problem is explained in this Section. In this work, we use MLP (modified to handle our data representation) as the base classifier. Here, we explain the s2sL approach by considering two-class classification task.\nData representation\nConsider a two-class classification task with INLINEFORM0 denoting the set of class labels, and let INLINEFORM1 and INLINEFORM2 be the number of samples corresponding to INLINEFORM3 and INLINEFORM4 , respectively. In general, to train a classifier, the samples in the train set are provided as input-output pairs as follows. DISPLAYFORM0\nwhere INLINEFORM0 refers to the INLINEFORM1 -dimensional feature vector representing the INLINEFORM2 sample corresponding to INLINEFORM3 class label, and INLINEFORM4 , refers to output label of INLINEFORM5 class. In the proposed data representation format, called simultaneous two sample (s2s) representation, we will simultaneously consider two samples as follows. DISPLAYFORM0\nwhere INLINEFORM0 , INLINEFORM1 are the INLINEFORM2 -dimensional feature vectors representing the INLINEFORM3 sample in INLINEFORM4 class and INLINEFORM5 sample in the INLINEFORM6 class, respectively; and INLINEFORM7 , INLINEFORM8 refers to the output labels of INLINEFORM9 and INLINEFORM10 class, respectively.\nHence, in s2s data representation, we will have an input feature vector of length INLINEFORM0 i.e., INLINEFORM1 , and output class labels as either INLINEFORM2 or INLINEFORM3 . Each sample can be combined with all the samples (i.e., with ( INLINEFORM4 ) samples) in the dataset. Therefore, by representing the data in the s2s format, the number of samples in the train set increases to INLINEFORM5 from INLINEFORM6 samples. We hypothesize that the s2s format is expected to help the network not only to learn the characteristics of the two classes separately, but also the difference and similarity in characteristics of the two classes.\nClassifier training\nMLP, the most commonly used feed forward neural network, is considered as the base classifier to validate our proposed s2s framework. Generally, MLPs are trained using the data format given by eq. INLINEFORM0 . But to train the MLP on our s2s based data representation (as in eq. INLINEFORM1 ), the following modifications are made to the MLP architecture (refer to Figure FIGREF4 ).\nWe have INLINEFORM0 units (instead of INLINEFORM1 units) in the input layer to accept the two samples i.e., INLINEFORM2 and INLINEFORM3 , simultaneously.\nThe structure of the hidden layer in this approach is similar to that of a regular MLP. The number of hidden layers and hidden units can be varied depending upon the complexity of the problem. The number of units in the hidden layer is selected empirically by varying the number of hidden units from 2 to twice the length of the input layer (i.e., 2 to INLINEFORM0 ) and the unit at which the highest performance is obtained are selected. In this paper, we considered only a single hidden layer. Rectified linear units (ReLU) are used for hidden layer.\nThe output layer will consist of units equal to twice the considered number of classes in the classification task i.e, the output layer will have four units for two-class classification task. The sigmoid activation function (not softmax) is used on the output layer units. Unlike regular MLP, we use sigmoid activation units in the output layer, with binary cross-entropy as the cost function, because the output labels in the proposed s2s based data representation will have more than one unit active at a time (not one-hot encoded output) and this condition cannot be handled using softmax function.\nAs can be seen from Figure FIGREF4 , the output layer in our proposed method has outputs INLINEFORM0 and INLINEFORM1 which corresponds to the outputs associated with the input feature vectors INLINEFORM2 and INLINEFORM3 , respectively. For a two-class classification problem, there will be four units in the output layer and the possible output labels are INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 corresponding to the class labels INLINEFORM8 and INLINEFORM9 , respectively. This architecture is referred to as s2s-MLP. In s2sL, s2s-MLP is trained using the s2s data representation format. Further, the s2s-MLP is trained using adam optimizer.\nClassifier testing\nGenerally, the feature vector corresponding to the test sample is provided as input to the trained MLP in the testing phase and the class label is decided based on the obtained output.\nHowever, in s2sL method, the feature vector corresponding to the test sample should also be converted to the s2s data representation format to test the trained s2s-MLP. We propose a testing approach, where the given test sample is combined with a set of preselected reference samples, whose class label is known a priori, to generate multiple instances of the same test sample as follows. DISPLAYFORM0\nwhere INLINEFORM0 , INLINEFORM1 refer to the INLINEFORM2 -dimensional feature vector corresponding to the test sample and the INLINEFORM3 reference sample, respectively. INLINEFORM4 refers to the considered number of reference samples. These reference samples can be selected from any of the two classes.\nFor testing the s2s-MLP (as shown in Figure FIGREF4 ), each test sample INLINEFORM0 (same as ' INLINEFORM1 ' in (3)) is combined with all the INLINEFORM2 reference samples ( INLINEFORM3 ) to form INLINEFORM4 different instances of the same test sample INLINEFORM5 . The corresponding outputs ( INLINEFORM6 ) obtained from s2s-MLP for the INLINEFORM7 generated instances of INLINEFORM8 are combined by voting-based decision approach to obtain the final decision INLINEFORM9 . The class label that gets maximum votes is considered as the predicted output label.\nExperiments\nWe validate the performance of the proposed s2sL by providing the preliminary results obtained on two different tasks namely, Speech/Music discrimination and emotion classification. We considered the GTZAN Music-Speech dataset [17], consisting of 120 audio files (60 speech and 60 music), for task of classifying speech and music. Each audio file (of 2 seconds duration) is represented using a 13-dimensional mel-frequency cepstral coefficient (MFCC) vector, where each MFCC vector is the average of all the frame level (frame size of 30 msec and an overlap of 10 msec) MFCC vectors. It is to be noted that our main intention for this task is not better feature selection, but to demonstrate the effectiveness of our approach, in particular for low data scenarios.\nThe standard Berlin speech emotion database (EMO-DB) [18] consisting of 535 utterances corresponding to 7 different emotions is considered for the task of emotion classification. Each utterance is represented by a 19-dimensional feature vector obtained by using the feature selection algorithm from WEKA toolkit [19] on the 384-dimensional utterance level feature vector obtained using openSMILE toolkit [20]. For two class classification, we considered the two most confusing emotion pairs i.e., (Neutral,Sad) and (Anger, Happy). Data corresponding to Speech/Music classification (60 speech and 60 music samples) and Neutral/Sad classification (79 neutral and 62 sad utterances) is balanced whereas Anger/Happy classification task has data imbalance, with anger forming the majority class (127 samples) and happy forming the minority class (71 samples). Therefore, we show the performance of s2sL on both, balanced and imbalanced datasets.\nAll experimental results are validated using 5-fold cross validation (80% of data for training and 20% for testing in each fold). Further, to analyze the effectiveness of s2sL in low resource scenarios, different proportions of training data, within each fold, are considered to train the system. For this analysis, we considered 4 different proportions i.e., INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 of the training data to train the classifier. For instance, INLINEFORM4 means considering only half of the original training data to train the classifier, and INLINEFORM5 means considering the complete training data. 5-fold cross validation is considered for all data proportions. Accuracy (in %) is used as a performance measure for balanced data classification tasks (i.e., Speech/Music classification and Neutral/Sad emotion classification), whereas the more preferred INLINEFORM6 measure [21] is used as a measure for imbalanced data classification task (i.e., Anger/Happy emotion classification).\nTable TABREF14 show the results obtained for proposed s2sL approach in comparison to that of MLP for the tasks of Speech/Music and Neutral/Sad classification, by considering different proportions of training data. The values in Table TABREF14 are mean accuracies (in %) obtained by 5-fold cross validation. It can be observed from Table TABREF14 that for both tasks, s2sL method outperforms MLP, especially at low resource conditions. s2sL shows an absolute improvement in accuracy of INLINEFORM0 % and INLINEFORM1 % over MLP for Speech/Music and Neutral/Sad classification tasks, respectively, when INLINEFORM2 of the original training data is used in experiments.\nTable TABREF14 show the results (in terms of INLINEFORM0 values) obtained for proposed s2sL approach in comparison to that of MLP for Anger/Happy classification (data imbalance problem). Here, state-of-the-art methods i.e., Eusboost [22] and MWMOTE [23] are also considered for comparison. It can be observed from Table TABREF14 that the s2sL method outperforms MLP, and also performs better than Eusboost and MWMOTE techniques on imbalanced data (around 3 % absolute improvement in INLINEFORM1 value for s2sL compared to MWMOTE, when INLINEFORM2 of the training data is considered). In particular, at lower amounts of training data, s2sL outperforms all the other methods, illustrating its effectiveness even for low resourced data imbalance problems. s2sL method shows an absolute improvement of 6% ( INLINEFORM3 ) in INLINEFORM4 value over the second best ( INLINEFORM5 for MWMOTE), when only INLINEFORM6 of the training data is used.\nConclusions\nIn this paper, we introduced a novel s2s framework to effectively learn the class discriminative characteristics, even from low data resources. In this framework, more than one sample (here, two samples) are simultaneously considered to train the classifier. Further, this framework allows to generate multiple instances of the same test sample, by considering preselected reference samples, to achieve a more profound decision making. We illustrated the significance of our approach by providing the experimental results for two different tasks namely, speech/music discrimination and emotion classification. Further, we showed that the s2s framework can also handle the low resourced data imbalance problem.\nReferences\n[1] Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A. R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N. & Kingsbury, B. (2012) Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, pp. 82–97.\n[2] Vinyals, O., Ravuri, S. V. & Povey, D. (2012) Revisiting recurrent neural networks for robust ASR. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4085–4088.\n[3] Lu, L., Zhang, X., Cho K. & Renals, S. (2015) A study of the recurrent neural network encoder-decoder for large vocabulary speech recognition. In Proc. INTERSPEECH, pp. 3249–3253.\n[4] Zhang, Y., Pezeshki, M., Brakel, P., Zhang, S., Bengio, C.L.Y. & Courville, A. (2017) Towards end-to-end speech recognition with deep convolutional neural networks. arXiv preprint, arXiv:1701.02720.\n[5] Han, K., Yu, D. & Tashev, I. (2014) Speech emotion recognition using deep neural network and extreme learning machine. In Proc. Interspeech, pp. 223–227.\n[6] Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2016) Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network. In Proc. ICASSP, pp. 5200–5204).\n[7] Huang, Che-Wei, & Shrikanth S. Narayanan. (2016) Attention Assisted Discovery of Sub-Utterance Structure in Speech Emotion Recognition. In Proc. INTERSPEECH, pp. 1387–1391.\n[8] Zhang, Z., Ringeval, F., Han, J., Deng, J., Marchi, E. & Schuller, B. (2016) Facing realism in spontaneous emotion recognition from speech: Feature enhancement by autoencoder with LSTM neural networks. In Proc. INTERSPEECH, pp. 3593–3597.\n[9] Scheirer, E. D. & Slaney, M. (2003) Multi-feature speech/music discrimination system. U.S. Patent 6,570,991.\n[10] Pikrakis, A. & Theodoridis S. (2014) Speech-music discrimination: A deep learning perspective. In Proc. European Signal Processing Conference (EUSIPCO), pp. 616–620.\n[11] Choi, K., Fazekas, G. & Sandler, M. (2016) Automatic tagging using deep convolutional neural networks. arXiv preprint, arXiv:1606.00298.\n[12] Zazo Candil, R., Sainath, T.N., Simko, G. & Parada, C. (2016) Feature learning with raw-waveform CLDNNs for Voice Activity Detection. In Proc. Interspeech, pp. 3668–3672.\n[13] Tian, L., Moore, JD. & Lai, C. (2015) Emotion recognition in spontaneous and acted dialogues. In Proc. International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 698–704.\n[14] Dumpala, S.H., Chakraborty, R. & Kopparapu, S.K. (2017) k-FFNN: A priori knowledge infused feed-forward neural networks. arXiv preprint, arXiv:1704.07055.\n[15] Dumpala, S.H., Chakraborty, R. & Kopparapu, S.K. (2018) Knowledge-driven feed-forward neural network for audio affective content analysis. To appear in AAAI-2018 workshop on Affective Content Analysis.\n[16] Deng, J., Zhang, Z., Marchi, E. & Schuller, B. (2013) Sparse autoencoder-based feature transfer learning for speech emotion recognition. In Proc. Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), pp. 511–516.\n[17] George Tzanetakis, Gtzan musicspeech. Availabe on-line at http://marsyas.info/download/datasets.\n[18] Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W.F. & Weiss, B. (2005) A database of german emotional speech. In Proc. Interspeech, pp. 1517–1520.\n[19] Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P. & Witten, I.H. (2009) The WEKA data mining software: an update. ACM SIGKDD explorations newsletter, 11(1), pp. 10–18.\n[20] Eyben, F., Weninger, F., Gross, F. & Schuller, B. (2013) Recent developments in opensmile, the munich open-source multimedia feature extractor. In Proc. ACM international conference on Multimedia, pp. 835–838.\n[21] Maratea, A., Petrosino, A. & Manzo, M. (2014) Adjusted f-measure and kernel scaling for imbalanced data learning. Information Sciences 257:331–341.\n[22] Galar, M., Fernandez, A., Barrenechea, E. & Herrera, F. (2013) Eusboost: Enhancing ensembles for highly imbalanced data-sets by evolutionary undersampling. Pattern Recognition 46(12):3460–3471.\n[23] Barua, S., Islam, M. M., Yao, X. & Murase, K. (2014) Mwmote–majority weighted minority oversampling technique for imbalanced data set learning. IEEE Transactions on Knowledge and Data Engineering 26(2):405–425.",6 "char_len": 16622,7 "document_hash": "0898527b4e47c70f912ad3ee962a1c148330ba91abdeb24c1e9af78f46a3d1f6",8 "questions": [9 {10 "question_id": "q00",11 "qtype": "qasper",12 "question": "Which models/frameworks do they compare to?"13 }14 ]15}