A voice-based real-time emotion detection technique using recurrent neural network empowered feature modelling

Abstract

The advancements of the Internet of Things (IoT) and voice-based multimedia applications have resulted in the generation of big data consisting of patterns, trends and associations capturing and representing many features of human behaviour. The latent representations of many aspects and the basis of human behaviour is naturally embedded within the expression of emotions found in human speech. This signifies the importance of mining audio data collected from human conversations for extracting human emotion. Ability to capture and represent human emotions will be an important feature in next-generation artificial intelligence, with the expectation of closer interaction with humans. Although the textual representations of human conversations have shown promising results for the extraction of emotions, the acoustic feature-based emotion detection from audio still lags behind in terms of accuracy. This paper proposes a novel approach for feature extraction consisting of Bag-of-Audio-Words (BoAW) based feature embeddings for conversational audio data. A Recurrent Neural Network (RNN) based state-of-the-art emotion detection model is proposed that captures the conversation-context and individual party states when making real-time categorical emotion predictions. The performance of the proposed approach and the model is evaluated using two benchmark datasets along with an empirical evaluation on real-time prediction capability. The proposed approach reported 60.87% weighted accuracy and 60.97% unweighted accuracy for six basic emotions for IEMOCAP dataset, significantly outperforming current state-of-the-art models.

DeepEmoVision: Unveiling Emotion Dynamics in Video Through Deep Learning Algorithms

“…First, we use an annotated Twitter dataset to investigate relationships between emotion labels, social ties, and temporal patterns. Chen, L. et al [15] gives a summary of current work in the rapidly developing topic of automatic group emotion identification is given in this article. Research has used a variety of datasets, modalities (video, pictures, social media posts, audio), and approaches to investigate emotion analysis in crowds or groups.…”

“…The output of each layer is determined by multiplying its predecessor's output by the learnable weights of that layer is a kind of neural network architecture made to manage sequential data by preserving a hidden state that records details about the sequence's earlier inputs. RNNs are very helpful in Natural Language Processing (NLP) applications involving language production and understanding, where word or character order is important [15] Here is an overview of the main ideas behind recurrent neural networks in natural language processing: Processing Sequential Data: RNNs work well when processing data in sets, such time series or sentences. They work on a single sequence element at a time, keeping a concealed state that contains data from earlier components.…”

A cloud-based architecture for explainable Big Data analytics using self-structuring Artificial Intelligence

“…O'Leary explores some of the way Artificial Intelligence can be used to facilitate process and analyse Big Data in this manner [11]. This intersection of Big Data Analytics and AI is demonstrated in a number of recent studies, such as, situational awareness from IoT data streams [12], intelligent detection of driver behavior changes [13], human activity recognition [14] and emotion detection [15]. However, in most such studies a cloud based strategy is not proposed and the solution does not focus on the seamless integration of real time and batch processed data, further they also do not address the need for the explainability of the insights generated.…”

Emotion Intensity Detection in Online Media: An Attention Mechanism Based Multimodal Deep Learning Approach

“…How to effectively utilize the correlation between different modalities is a challenge. Appropriate model structures and algorithms need to be designed to capture and utilize inter-modal correlation information [8,9].…”