Using Hybrid HMM/DNN Embedding Extractor Models in Computational Paralinguistic Tasks

Abstract

The field of computational paralinguistics emerged from automatic speech processing, and it covers a wide range of tasks involving different phenomena present in human speech. It focuses on the non-verbal content of human speech, including tasks such as spoken emotion recognition, conflict intensity estimation and sleepiness detection from speech, showing straightforward application possibilities for remote monitoring with acoustic sensors. The two main technical issues present in computational paralinguistics are (1) handling varying-length utterances with traditional classifiers and (2) training models on relatively small corpora. In this study, we present a method that combines automatic speech recognition and paralinguistic approaches, which is able to handle both of these technical issues. That is, we trained a HMM/DNN hybrid acoustic model on a general ASR corpus, which was then used as a source of embeddings employed as features for several paralinguistic tasks. To convert the local embeddings into utterance-level features, we experimented with five different aggregation methods, namely mean, standard deviation, skewness, kurtosis and the ratio of non-zero activations. Our results show that the proposed feature extraction technique consistently outperforms the widely used x-vector method used as the baseline, independently of the actual paralinguistic task investigated. Furthermore, the aggregation techniques could be combined effectively as well, leading to further improvements depending on the task and the layer of the neural network serving as the source of the local embeddings. Overall, based on our experimental results, the proposed method can be considered as a competitive and resource-efficient approach for a wide range of computational paralinguistic tasks.

Research on Pig Sound Recognition Based on Deep Neural Network and Hidden Markov Models

“…A deep neural network (DNN) is a traditional feed-forward artificial neural network with multi-layer hidden units [ 26 ]. In the DNN-HMM framework, HMM is used to describe the dynamic changes of sound signals while DNN is used to estimate the probability of observed features [ 27 , 28 , 29 ].…”

Special Issue on Acoustic Sensors and Their Applications (Vol. 1)

“…Vetrab and Gosztolya [ 7 ] used hybrid HMM/DNN embedding extractor models in computational para-linguistic tasks. The proposed HMM/DNN hybrid acoustic-model-based feature extraction technique was then found efficient at extracting features from different para-linguistic tasks.…”