SimEmotion: A Simple Knowledgeable Prompt Tuning Method for Image Emotion Classification

Deng, Sinuo; Shi, Ge; Wu, Lifang; Xing, Lehao; Hu, Wenjin; Zhang, Heng; Xiang, Ye

doi:10.1007/978-3-031-00129-1_15

Cited by 4 publications

(4 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They showed that their prompts elicited more accurate factual knowledge from PLMs than the manually created prompts. Deng et al (2022a) presented a prompt-based fine-tuning strategy to learn task-specific sentiment representations while preserving knowledge contained in CLIP, resulting in a conceptually simple but empirically powerful framework for supervised image emotion classification. Zhou et al (2023) designed two consistency training strategies for prompt learning and conducted experiments on two multi-label emotion classification datasets.…”

Section: Prompt Learningmentioning

confidence: 99%

Multi-Modal Attentive Prompt Learning for Few-shot Emotion Recognition in Conversations

Liang,

Tu,

et al. 2024

jair

View full text Add to dashboard Cite

Emotion recognition in conversations (ERC) has emerged as an important research area in Natural Language Processing and Affective Computing, focusing on accurately identifying emotions within the conversational utterance. Conventional approaches typically rely on labeled training samples for fine-tuning pre-trained language models (PLMs) to enhance classification performance. However, the limited availability of labeled data in real-world scenarios poses a significant challenge, potentially resulting in diminished model performance. In response to this challenge, we present the Multi-modal Attentive Prompt (MAP) learning framework, tailored specifically for few-shot emotion recognition in conversations. The MAP framework consists of four integral modules: multi-modal feature extraction for the sequential embedding of text, visual, and acoustic inputs; a multi-modal prompt generation module that creates six manually-designed multi-modal prompts; an attention mechanism for prompt aggregation; and an emotion inference module for emotion prediction. To evaluate our proposed model’s efficacy, we conducted extensive experiments on two widely recognized benchmark datasets, MELD and IEMOCAP. Our results demonstrate that the MAP framework outperforms state-of-the-art ERC models, yielding notable improvements of 3.5% and 0.4% in micro F1 scores. These findings highlight the MAP learning framework’s ability to effectively address the challenge of limited labeled data in emotion recognition, offering a promising strategy for improving ERC model performance.

show abstract

Section: Prompt Learningmentioning

confidence: 99%

Multi-Modal Attentive Prompt Learning for Few-shot Emotion Recognition in Conversations

Liang,

Tu,

et al. 2024

jair

View full text Add to dashboard Cite

show abstract

“…Given the abstract nature of image emotions, it is difficult to obtain sufficient discriminative features from the image itself. However, to improve classification performance, some effort was made to enrich feature representations by incorporating external knowledge, such as proposing a well-designed sentiment dictionary [29][30][31], combining different types of dataset-specific information [32][33][34], or introducing different affective-specific knowledge [8,[35][36][37].…”

Section: Image Emotion Classificationmentioning

confidence: 99%

“…Researchers in visual emotion analysis classify emotions differently according to the categories or dimensions [7]. The category aspect includes descriptions, such as dominant categories [8] and category distributions [9]. Although methods for distribution and temporal-order prediction exist, currently, the most important and crucial task in visual emotion analysis is image emotion classification [2].…”

Section: Introductionmentioning

confidence: 99%

“…Adapting these recent techniques to IEC can be extremely valuable because human emotion is highly abstractive, and determining the emotion conveyed in images requires a deep understanding of various details and concepts; however, traditional vision pretraining methods can only access a fixed set of concepts [7]. Therefore, CLIP is a reasonable choice for IEC tasks [8].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning to compose diversified prompts for image emotion classification

Deng,

Wu,

Shi

et al. 2024

Comp. Visual Media

Self Cite

View full text Add to dashboard Cite

Image emotion classification (IEC) aims to extract the abstract emotions evoked in images. Recently, language-supervised methods such as contrastive language-image pretraining (CLIP) have demonstrated superior performance in image understanding. However, the underexplored task of IEC presents three major challenges: a tremendous training objective gap between pretraining and IEC, shared suboptimal prompts, and invariant prompts for all instances. In this study, we propose a general framework that effectively exploits the language-supervised CLIP method for the IEC task. First, a prompt-tuning method that mimics the pretraining objective of CLIP is introduced, to exploit the rich image and text semantics associated with CLIP. Subsequently, instance-specific prompts are automatically composed, conditioning them on the categories and image content of instances, diversifying the prompts, and thus avoiding suboptimal problems. Evaluations on six widely used affective datasets show that the proposed method significantly outperforms state-of-the-art methods (up to 9.29% accuracy gain on the EmotionROI dataset) on IEC tasks with only a few trained parameters. The code is publicly available at https://github.com/dsn0w/PT-DPC/for research purposes.

show abstract

A Domain-aware Language-supervised Method for Image Emotion Classification

Deng,

Shi,

et al. 2023

2023 IEEE 4th International Conference on Pattern Recognition and Machine Learning (PRML)

View full text Add to dashboard Cite

SimEmotion: A Simple Knowledgeable Prompt Tuning Method for Image Emotion Classification

Cited by 4 publications

References 10 publications

Multi-Modal Attentive Prompt Learning for Few-shot Emotion Recognition in Conversations

Multi-Modal Attentive Prompt Learning for Few-shot Emotion Recognition in Conversations

Learning to compose diversified prompts for image emotion classification

A Domain-aware Language-supervised Method for Image Emotion Classification

Contact Info

Product

Resources

About