Speech Emotion Recognition Using Deep Neural Network Considering Verbal and Nonverbal Speech Sounds

Huang, Kun-Yi; Wu, Chung-Hsien; Hong, Qian-Bei; Su, Ming-Hsiang; Chen, Yi‐Hsuan

doi:10.1109/icassp.2019.8682283

Cited by 86 publications

(44 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Huang's approach, CNN was trained both on verbal and nonverbal segments of speech and CNN learned features were used by LSTM for recognizing speech emotions [219].…”

Section: Cnns Based Speech Recognitionmentioning

confidence: 99%

A survey of the recent architectures of deep convolutional neural networks

et al. 2020

View full text Add to dashboard Cite

Deep Convolutional Neural Networks (CNNs) are a special type of Neural Networks, which have shown exemplary performance on several competitions related to Computer Vision and Image Processing. Interesting application areas of CNN include Image Classification and Segmentation, Object Detection, Video Processing, Natural Language Processing, Speech Recognition, etc. The powerful learning ability of deep CNN is largely due to the use of multiple feature extraction stages that can automatically learn representations from the data. Availability of a large amount of data and improvements in the hardware technology have accelerated the research in CNNs, and recently very interesting deep CNN architectures have been reported. In fact, several interesting ideas to bring advancements in CNNs have been explored such as the use of different activation and loss functions, parameter optimization, regularization, and architectural innovations. However, the major improvement in representational capacity of the deep CNN is achieved through architectural innovations. Especially, the idea of exploiting spatial and channel information, depth and width of architecture, and multi-path information processing has gained substantial attention. Similarly, the idea of using a block of layers as a structural unit is also gaining popularity. This survey thus focuses on the intrinsic taxonomy present in the recently reported deep CNN architectures and consequently, classifies the recent innovations in CNN architectures into seven different categories. These seven categories are based on spatial exploitation, depth, multi-path, width, feature-map exploitation, channel boosting, and attention. Additionally, the elementary understanding of CNN components, current challenges and applications of CNN are also provided. CNNs are the best among learning algorithms in understanding images content, and have shown exemplary results in segmentation, classification, detection, and retrieval related tasks [8], [9]. The success of CNNs has captured attention beyond academia. In industry, companies such as Google, Microsoft, AT&T, NEC, and Facebook have developed active research groups for exploring new architectures of CNN [10]. At present, most of the frontrunners of image processing and computer vision competitions are employing deep CNN based models.The attractive feature of CNN is its ability to exploit spatial or time correlation of the data. The topology of CNN is divided into multiple learning stages composed of a combination of the convolutional layers, non-linear processing units, and subsampling layers [11]. CNNs are feedforward multilayered hierarchical networks that are similar to fully connected neural network where each layer, using a bank of convolutional kernels, performs multiple transformations [12]. Convolution operation extracts useful features from locally correlated data points. Output of the convolutional kernels is assigned to non-linear processing unit (activation function), which not only helps in learning abstractions but also emb...

show abstract

“…Huang's approach, CNN was trained both on verbal and nonverbal segments of speech and CNN learned features were used by LSTM for recognizing speech emotions [219].…”

Section: Cnns Based Speech Recognitionmentioning

confidence: 99%

A survey of the recent architectures of deep convolutional neural networks

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The late-fusion of facial expressions, speech signals, and text information reached the third-best result, with a CCC of 0.27 for arousal and 0.35 for valence. The complex attention-based network proposed by Huang et al [10] was able to achieve a CCC of 0.31 in arousal and 0.45 in valence, using only visual information.…”

Section: Omg-emotionmentioning

confidence: 99%

“…Deep learning models usually learn how to represent affective features by updating filters based on a large number of data samples, using strongly supervised learning methods [9][10][11][12][13][14]. As a result, these models can extract facial features for a collection of different individuals, which contributes to their generalization of expression representations enabling a universal and automatic FER machine.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The FaceChannel: A Fast and Furious Deep Neural Network for Facial Expression Recognition

2020

View full text Add to dashboard Cite

Current state-of-the-art models for automatic facial expression recognition (FER) are based on very deep neural networks that are effective but rather expensive to train. Given the dynamic conditions of FER, this characteristic hinders such models of been used as a general affect recognition. In this paper, we address this problem by formalizing the FaceChannel, a light-weight neural network that has much fewer parameters than common deep neural networks. We introduce an inhibitory layer that helps to shape the learning of facial features in the last layer of the network and, thus, improving performance while reducing the number of trainable parameters. To evaluate our model, we perform a series of experiments on different benchmark datasets and demonstrate how the FaceChannel achieves a comparable, if not better, performance to the current state-of-the-art in FER. Our experiments include cross-dataset analysis, to estimate how our model behaves on different affective recognition conditions. We conclude our paper with an analysis of how FaceChannel learns and adapts the learned facial features towards the different datasets.

show abstract

“…Since the output value of the nerve cell exceeds the appropriate range of the activation function itself before passing the activation function, the nerve cell may fail to work [30]. In order to solve this problem, this article introduces the bn [31], the batch normalization method is as follows:…”

Section: B Lstm-based Emotion Fusion and Recognitionmentioning

confidence: 99%

Dance Emotion Recognition Based on Laban Motion Analysis Using Convolutional Neural Network and Long Short-Term Memory

Wang

Cao

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Dance emotion recognition technology is of great significance for the digitalization, virtual performance, inheritance and protection of folk dance. Based on the mechanism that emotion expression in dance performance can be fully expressed through the strength and rhythm of dance movements, a novel dance emotion expression method is proposed to train hybrid deep learning neural network, to effectively identify the seven basic dance emotions of fear, anger, boredom, excitement, joy, relaxation and sadness. First, in order to fully express the emotions contained in the dance movements, this paper defines a dance emotion expression method through Laban Movement Analysis (LMA) method, which includes the characteristic parameters of the three aspects of body structure, spatial orientation and force effect, and converts the original dance movement data into three characteristic expression parameters to obtain dance emotion data. Then, Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) hybrid neural network models are used to test and train dance emotion data. Finally, in order to verify the applicability of the CNN-LSTM model, decision tree, random forest, CNN and LSTM are established and compared for accuracy. The results show that it is feasible to identify dance emotion from the perspective of dance movement, and the CNN-LSTM model is of high accuracy.

show abstract

Speech Emotion Recognition Using Deep Neural Network Considering Verbal and Nonverbal Speech Sounds

Cited by 86 publications

References 18 publications

A survey of the recent architectures of deep convolutional neural networks

A survey of the recent architectures of deep convolutional neural networks

The FaceChannel: A Fast and Furious Deep Neural Network for Facial Expression Recognition

Dance Emotion Recognition Based on Laban Motion Analysis Using Convolutional Neural Network and Long Short-Term Memory

Contact Info

Product

Resources

About