Evaluating Significant Features in Context-Aware Multimodal Emotion Recognition with XAI Methods

Shaikh, Talal; Khalane, Aaishwarya; Makwana, Rikesh; Ullah, Abrar

doi:10.22541/au.167407909.97031004/v1

Cited by 3 publications

(2 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These approaches have been deployed in almost every category of the novel taxonomy we propose above. For example, [125] apply gra-dientSHAP to a multimodal model using BERT for emotion classification. In news classification, [126] have added LIME to a BERT-based classifier detecting misinformation about COVID-19, which show the users how the decision was reached which data sources were used to make the classification, extracting sentences from relative news articles to explain the classification.…”

Section: How Safe?mentioning

confidence: 99%

A Survey of Text Classification With Transformers: How Wide? How Large? How Long? How Accurate? How Expensive? How Safe?

Fields,

Chovanec,

Madiraju

2024

IEEE Access

View full text Add to dashboard Cite

Text classification is a basic task in natural language processing (NLP) with applications from sentiment analysis to question-answering with chat bots. In recent years, transformer-based models have emerged as the prevailing framework in NLP, demonstrating excellent results across many benchmarks. This paper recommends an expanded taxonomy of applications and provides a review of the performance of different models across these applications. The use of traditional research techniques plus co-citation and bibliographic coupling provides a comprehensive view of the current and past research in this area. The study begins by providing an overview of the history of transformer-based models with an emphasis on recent large language models (LLM). Next, uni-modal (text only) inputs and the emerging area of multi-modal classification are discussed to provide a comparison of current and emerging research in this area. Gaps are highlighted in the use of multi-modal text/numeric/columnar data and recommendations for future research are provided. Finally, the length of text input variables (tokens) is reviewed to explore the evolution from short-text to longer document applications. Furthermore, the accuracy on 358 datasets across 20 applications is reviewed and unexpected results emerge which show that LLMs are not always the most accurate or least expensive option. In addition to model performance, the safety implications of transformer-based models are reviewed, and a summary of issues related to ethics, bias, social implications, and copyright are explored.

show abstract

Section: How Safe?mentioning

confidence: 99%

A Survey of Text Classification With Transformers: How Wide? How Large? How Long? How Accurate? How Expensive? How Safe?

Fields,

Chovanec,

Madiraju

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Multimodal analysis requires correlations and effective Unimodal features, unlike Unimodal issues. The study [38] examined the need for human-understandable interpretations of black-box DL models are highlighted in this paper, which focuses on the hazards of employing these algorithms for vital tasks like emotion identification.…”

Section: Literature Reviewmentioning

confidence: 99%

TER-CA-WGNN: Trimodel Emotion Recognition Using Cumulative Attribute-Weighted Graph Neural Network

Al-Saadawi,

Das

2024

Applied Sciences

View full text Add to dashboard Cite

Affective computing is a multidisciplinary field encompassing artificial intelligence, natural language processing, linguistics, computer science, and social sciences. This field aims to deepen our comprehension and capabilities by deploying inventive algorithms. This article presents a groundbreaking approach, the Cumulative Attribute-Weighted Graph Neural Network, which is innovatively designed to integrate trimodal textual, audio, and visual data from the two multimodal datasets. This method exemplifies its effectiveness in performing comprehensive multimodal sentiment analysis. Our methodology employs vocal inputs to generate speaker embeddings trimodal analysis. Using a weighted graph structure, our model facilitates the efficient integration of these diverse modalities. This approach underscores the interrelated aspects of various emotional indicators. The paper’s significant contribution is underscored by its experimental results. Our novel algorithm achieved impressive performance metrics on the CMU-MOSI dataset, with an accuracy of 94% and precision, recall, and F1-scores above 92% for Negative, Neutral, and Positive emotion categories. Similarly, on the IEMOCAP dataset, the algorithm demonstrated its robustness with an overall accuracy of 93%, where exceptionally high precision and recall were noted in the Neutral and Positive categories. These results mark a notable advancement over existing state-of-the-art models, illustrating the potential of our approach in enhancing Sentiment Recognition through the synergistic use of trimodal data. This study’s comprehensive analysis and significant results demonstrate the proposed algorithm’s effectiveness in nuanced emotional state recognition and pave the way for future advancements in affective computing, emphasizing the value of integrating multimodal data for improved accuracy and robustness.

show abstract

Explainable Artificial Intelligence (XAI) for Emotion Detection

Ayyalasomayajula,

Pandey

2024

Advances in Psychology, Mental Health, and Behavioral Studies

View full text Add to dashboard Cite

This chapter delves into the significance of explainable artificial intelligence (XAI) in emotion detection (ED) systems, which aim to provide transparency and interpretability in affective computing. The chapter introduces ED systems, defining their purpose and importance in various industries. Subsequently, the need for XAI in emotion detection is discussed, emphasizing ethical concerns, legal requirements, and user trust. Next, the fundamentals of ED systems are explored, encompassing techniques for emotion recognition via facial expressions, voice tones, and text. The challenges associated with these techniques, including variability in human expressions, cultural differences, and data scarcity, are addressed. Next, explanation methods for ED models, and the popular XAI frameworks are presented and evaluated. Quantitative and qualitative evaluation metrics are employed to assess the effectiveness of XAI in ED. Lastly, three case studies demonstrate the successful application of XAI, and as research evolves, future directions that include advanced explainable ED are discussed.

show abstract

Evaluating Significant Features in Context-Aware Multimodal Emotion Recognition with XAI Methods

Cited by 3 publications

References 44 publications

A Survey of Text Classification With Transformers: How Wide? How Large? How Long? How Accurate? How Expensive? How Safe?

A Survey of Text Classification With Transformers: How Wide? How Large? How Long? How Accurate? How Expensive? How Safe?

TER-CA-WGNN: Trimodel Emotion Recognition Using Cumulative Attribute-Weighted Graph Neural Network

Explainable Artificial Intelligence (XAI) for Emotion Detection

Contact Info

Product

Resources

About