Multi-Label Multimodal Emotion Recognition With Transformer-Based Fusion and Emotion-Level Representation Learning

Le, Hoai-Duy; Lee, Guee-Sang; Kim, Soo-Hyung; Kim, Seungwon; Yang, Hyung-Jeong

doi:10.1109/access.2023.3244390

Cited by 23 publications

(8 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Transformers are one of the techniques used in the Deep learning framework. The tex-tual data can be processed by the transformers for recognizing the emotions [31], [32]. Transformers are used in sequence transduction.…”

Section: Deep Learning Techniques Used For Emotion Detectionmentioning

confidence: 99%

“…Transformers: [76] showed Emotion Recognition from Videos Using Transformer Models. [32] presented transformer-based fusion and representation learning method to fuse and enrich multimodal features from raw videos for the task of multi-label video emotion recognition. [77] presented bi-modal transformer.…”

Section: Machine Learning Techniques Used For Emotion Detectionmentioning

confidence: 99%

See 1 more Smart Citation

A Survey on Multi-modal Emotion Detection Techniques

Chatterjee,

Shah,

Bhatt

et al. 2024

Preprint

View full text Add to dashboard Cite

The utilization of emotion detection and recognition technologies has revolution- ized human-computer interactions in various fields such as sentiment analysis, health monitoring, education, and automotive interfaces. Previously, traditional systems relied on single-channel affect sensing, which limited their ability to cap- ture the complexity of human emotions. However, humans naturally combine multiple cues such as facial expressions, speech, gestures, and contextual factors when expressing their emotions. As a result, there has been a growing inter- est in multi-modal emotion frameworks that integrate different sensory streams to obtain more comprehensive emotion assessments. These holistic perspectives allow for the capture of nuanced affective information that would otherwise be difficult to represent. In this survey paper, we delve into the latest advancements in emotion recognition systems, examining fusion techniques, feature engineer- ing methods, and classification architectures that leverage inputs from various modalities such as vision, audio, and text. Our focus is to showcase innova- tive interventions throughout the entire pipeline, from preprocessing raw signals to predicting emotion labels, in order to enable robust multi-modal analysis. Through detailed theoretical discussions and practical case studies, this paper aims to inspire further research by providing insights into the current state-of- the-art, highlighting open challenges, and exploring promising avenues in emotion detection through cross-modal learning.

show abstract

Section: Deep Learning Techniques Used For Emotion Detectionmentioning

confidence: 99%

Section: Machine Learning Techniques Used For Emotion Detectionmentioning

confidence: 99%

A Survey on Multi-modal Emotion Detection Techniques

Chatterjee,

Shah,

Bhatt

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…where, 𝑦 𝑠 represents the true labels, and 𝑦 𝑠 ̂ represents the correct label predicted by the model. The F-score is a composite metric that is a weighted harmonic average of precision and recall [30]. The F1 score is based on a harmonic average of precision and recall, calculated by:…”

Section: Evaluation Of the Proposed Systemmentioning

confidence: 99%

Multilingual sentiment analysis during the pandemic using deep learning models

Chang

2024

ACE

View full text Add to dashboard Cite

High mood swings and increased mortality with the COVID-19 outbreak, mood changes are a concern. COVID-19 is extremely contagious, and by December 20, 2021, the global death toll from COVID-19 exceeded 5.35 million. Under such circumstances, people changed their original way of life and isolated themselves from home to work online. Most people in this situation expressed their emotions through social media. In order to understand the changes of people's emotions during the pandemic, this research analyzed the emotions of people's messages posted on social media in different countries through the SenWave dataset by different deep learning models. The dataset contains 6 languages (English, Arabic, Spanish, French, Italian and Chinese) with more than 70,000 tweets, in this paper, only tweets collected from Twitter are concerned, which contains 11 labels. The dataset will be preprocessed and transferred into numerical vector, then TextCNN, BERT, and BERT-TextCNN models are implemented for the above problems. BERT model reached 0.69 and 0.53 on precision and recall respectively on Twitter dataset.

show abstract

“…Attention mechanisms and transformer architectures have then emerged as powerful techniques to achieve state-of-theart results in various natural language processing (NLP) tasks, including multi-label emotion classification [12], [20]. Transformers operate by employing self-attention mechanisms, capturing contextual relationships between tokens (words) in a sequence.…”

Section: A Multi-label Emotion Classificationmentioning

confidence: 99%

Pseudo-Labeling With Large Language Models for Multi-Label Emotion Classification of French Tweets

Malik,

Bernard,

Pauchet

et al. 2024

IEEE Access

View full text Add to dashboard Cite

This study proposes a novel semi-supervised multi-label emotion classification approach for French tweets based on pseudo-labeling. Human subjectivity in emotional expression makes it difficult for a machine to learn. Therefore, it necessitates training supervised learning models on large datasets annotated by multiple annotators. However, creating such datasets can be costly and time-consuming. Moreover, aggregating annotations from multiple annotators to capture their collective emotional state adds complexity to the task. Semi-supervised learning techniques have shown effectiveness with limited datasets. Furthermore, Large language Models (LLMs), particularly Chat-GPT, have demonstrated superior annotation accuracy compared to annotations obtained from crowdsourcing platforms, when both are evaluated against expertannotated data. This work introduces a novel approach for multi-label emotion classification of French tweets by leveraging pseudo-labels generated through Chat-GPT, a robust large language model. Using zero-shot, one-shot, and few-shot learning techniques, Chat-GPT annotates the unlabelled part of our dataset. These Chat-GPTannotated pseudo-labels are then merged with manual annotations, facilitating the training of a multilabel emotion classification model via semi-supervised learning. Furthermore, within the context of our research, we present a new French tweet dataset, containing testimonials from people affected by an urban industrial incident. This dataset features 2,350 tweets, each manually annotated by three human annotators based on 8 pre-identified emotions. Benchmark results are presented for multi-label emotion classification models employing both fully supervised and semi-supervised approaches with pseudo-labeling. Our findings demonstrate the superiority of our approach for multi-label emotion classification over standard pseudolabeling and an established baseline.

show abstract

Multi-Label Multimodal Emotion Recognition With Transformer-Based Fusion and Emotion-Level Representation Learning

Cited by 23 publications

References 35 publications

A Survey on Multi-modal Emotion Detection Techniques

A Survey on Multi-modal Emotion Detection Techniques

Multilingual sentiment analysis during the pandemic using deep learning models

Pseudo-Labeling With Large Language Models for Multi-Label Emotion Classification of French Tweets

Contact Info

Product

Resources

About