Mel-MViTv2: Enhanced Speech Emotion Recognition With Mel Spectrogram and Improved Multiscale Vision Transformers

Abstract

Speech emotion recognition aims to automatically identify and classify emotions from speech signals. It plays a crucial role in various applications such as human-computer interaction, affective computing, and social robotics. Over the years, researchers have proposed different approaches for speech emotion recognition, leveraging various classifiers and features. However, despite the advancements, existing methods in speech emotion recognition still have certain limitations. Some approaches rely on handcrafted features that may not capture the full complexity of emotional information present in speech signals, while others may suffer from a lack of robustness and generalization when applied to different datasets. To address these challenges, this paper proposes a speech emotion recognition method that combines Mel spectrogram with Short-Term Fourier Transform (Mel-STFT) and the Improved Multiscale Vision Transformers (MViTv2). The Mel-STFT spectrograms capture both the frequency and temporal information of speech signals, providing a more comprehensive representation of the emotional content. The MViTv2 classifier introduces multi-scale visual modeling with different stages and pooling attention mechanisms. MViTv2 incorporates relative positional embeddings and a residual pooling connection to effectively model the interactions between tokens in the space-time structure, preserve essential information, and improve the efficiency of the model. Experimental results demonstrate that the proposed method generalizes well on different datasets, achieving an accuracy of 91.51% on the Emo-DB dataset, 81.75% on the RAVDESS dataset, and 64.03% on the IEMOCAP dataset.

XEmoAccent: Embracing Diversity in Cross-Accent Emotion Recognition Using Deep Learning

“…These features serve as the The comprehensive benchmark studies in Table 9, with their respective subset of features, datasets, and accuracy scores, show a snapshot of the broader research landscape in SER. Multiple features based speech emotion recognition systems are proposed considering distinct machine learning models such as voting classifier [19], [61], attentionbased multi-learning model (ABMD) [23], 1D-CNN [26] and MViTv2 [60]. However, these multi-featured emotion recognition systems target a particular region accent.…”

Emotion Recognition and Multi-class Classification in Music with MFCC and Machine Learning

“…This technique visualizes the changes in frequency over time, aiding in understanding the complex structures within audio signals. Designed to mimic the characteristics of human hearing, the Mel scale processes frequencies akin to how humans perceive sound, sharing similarities with MFCC analysis but emphasizing the visual representation of audio signals [14]- [15].…”

Carrier-Free Ultra-Wideband Sensor Target Recognition in the Jungle Environment

“…Signal intra-pulse sequences are transformed into gray-scaled STFT spectrograms, and a CNN network is designed to extract features and classify the spectrogram [25]. Kah Liang Ong proposes a speech emotion recognition method that combines the Mel spectrogram with the Short-Term Fourier Transform (Mel-STFT) and Improved Multiscale Vision Transformers (MViTv2) [26]. However, Markov transfer field images possess several advantages in comparison with time-frequency maps and GAF transformations.…”