Bangla Speech Emotion Recognition and Cross-Lingual Study Using Deep CNN and BLSTM Networks

Abstract

In this study, we have presented a deep learning-based implementation for speech emotion recognition (SER). The system combines a deep convolutional neural network (DCNN) and a bidirectional long-short term memory (BLSTM) network with a time-distributed flatten (TDF) layer. The proposed model has been applied for the recently built audio-only Bangla emotional speech corpus SUBESCO. A series of experiments were carried out to analyze all the models discussed in this paper for baseline, cross-lingual, and multilingual training-testing setups. The experimental results reveal that the model with a TDF layer achieves better performance compared with other state-of-the-art CNN-based SER models which can work on both temporal and sequential representation of emotions. For the cross-lingual experiments, cross-corpus training, multi-corpus training, and transfer learning were employed for the Bangla and English languages using the SUBESCO and RAVDESS datasets. The proposed model has attained a state-of-the-art perceptual efficiency achieving weighted accuracies (WAs) of 86.9%, and 82.7% for the SUBESCO and RAVDESS datasets, respectively.

A Review of the Advancement in Speech Emotion Recognition for Indo-Aryan and Dravidian Languages

“…Te model implemented for the RAVDESS database outperforms the state-of-the-art accuracy giving 97.14% highest recognition rate [78]. Combining DCNN and BLSTM, the model proposed by Sultana et al [3] obtained state-of-the-art efciency with 82.7% and 86.9% accuracy for the RAVDESS and SUBESCO databases for English and Bangla languages, respectively.…”

“…Terefore, deep learning approaches have become more In recent times, multitask learning and attention mechanism are also being used for improved performance [58,59]. For cross-corpus and cross-lingual speech emotion recognition, the transfer learning technique is being widely used [3,60,61].…”

“…Experiments have also been conducted for cross-lingual speech emotion recognition. Sultana et al [3] showed a crosslingual study for English and Bangla languages using RAV-DESS and SUBESCO datasets, respectively, where the proposed system integrates a deep CNN and a BLSTM network with a TDF layer. Transfer learning was used for the crosslingual experiment, achieving weighted accuracy of 86.9% for SUBESCO and 82.7% for RAVDESS.…”

“…It has been seen from cross-lingual speech emotion recognition studies that models trained with a language corpus do not perform well when tested on a diferent language corpus compared to the monolingual recognition rate [2,3]. However, it will be interesting to fnd out whether those models perform better for diferent languages from the same groups.…”

Two-Way Feature Extraction for Speech Emotion Recognition Using Deep Learning

“…Apart from CNNs [ 13 , 14 , 15 ], Long Short-term Memory Networks (LSTM) [ 16 ], DNN [ 17 , 18 ] and CNN-LSTM hybrids have also shown promising results in the field of emotion recognition. LSTMs are more advanced Recurrent Neural Networks (RNN) optimized to use gates to control information flow.…”