Multi-Input Speech Emotion Recognition Model Using Mel Spectrogram and GeMAPS

Abstract

The existing research on emotion recognition commonly uses mel spectrogram (MelSpec) and Geneva minimalistic acoustic parameter set (GeMAPS) as acoustic parameters to learn the audio features. MelSpec can represent the time-series variations of each frequency but cannot manage multiple types of audio features. On the other hand, GeMAPS can handle multiple audio features but fails to provide information on their time-series variations. Thus, this study proposes a speech emotion recognition model based on a multi-input deep neural network that simultaneously learns these two audio features. The proposed model comprises three parts, specifically, for learning MelSpec in image format, learning GeMAPS in vector format, and integrating them to predict the emotion. Additionally, a focal loss function is introduced to address the imbalanced data problem among the emotion classes. The results of the recognition experiments demonstrate weighted and unweighted accuracies of 0.6657 and 0.6149, respectively, which are higher than or comparable to those of the existing state-of-the-art methods. Overall, the proposed model significantly improves the recognition accuracy of the emotion “happiness”, which has been difficult to identify in previous studies owing to limited data. Therefore, the proposed model can effectively recognize emotions from speech and can be applied for practical purposes with future development.

Enhancing Speech Emotion Recognition Using Dual Feature Extraction Encoders

“…The accurate and effective extraction of relevant characteristics, as well as the high correlation among these features, are critical elements that significantly affect the effectiveness of the emotion detection system. Contemporary SER approaches have been positively affected by the introduction of several innovative feature extraction methods [ 17 , 18 , 19 , 20 ]. In one study [ 17 ], a deep neural network model for SER that could simultaneously learn both MelSpec and GeMAPS audio features was proposed.…”

“…Contemporary SER approaches have been positively affected by the introduction of several innovative feature extraction methods [ 17 , 18 , 19 , 20 ]. In one study [ 17 ], a deep neural network model for SER that could simultaneously learn both MelSpec and GeMAPS audio features was proposed. The three components of the model are the learning of MelSpec in picture format, learning of GeMAPS in vector format, and combining the two to predict emotions.…”

Penetration State Identification of Aluminum Alloy Cold Metal Transfer Based on Arc Sound Signals Using Multi-Spectrogram Fusion Inception Convolutional Neural Network

“…Feature extraction is a critical step in classification and recognition using deep learning algorithms. The Mel filter bank, Gammatone filter bank and Bark filter bank are used to extract the spectrograms of sound signals in speech recognition and classification studies based on sound signals [24][25][26][27]. In order to extract richer features, a method for extracting fusion spectrograms is proposed, as shown in Figure 2.…”

Exploring the Effectiveness of the Phase Features on Double Compressed AMR Speech Detection

“…As a result, this study focuses on detecting double-compressed (DC) AMR speech signals. The magnitude of the discrete Fourier transform (DFT) of short speech segments, commonly known as the spectrogram representation of speech signals, has found widespread application in various tasks such as speaker recognition [6], speech recognition [7], emotion recognition [8], and audio event detection [9]. Its effectiveness stems from its ability to capture the spectral content variation of the signal over time, making it suitable for use with deep neural networks (DNNs), such as deep convolutional neural networks (CNNs), and long-short-term memory (LSTM) networks.…”