EMP: Emotion-guided Multi-modal Fusion and Contrastive Learning for Personality Traits Recognition

Wang, Yusong; Li, Dongyuan; Funakoshi, Kotaro; Okumura, Manabu

doi:10.1145/3591106.3592243

Cited by 5 publications

(3 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Han et al 13 created a multi‐party conversation‐based personality dataset derived from CPED, consisting of 1195 data samples for personality recognition, and introduced a speaker‐aware layering named SH‐Transformer converter. Wang et al 4 employed an emotion‐guided multi‐modal fusion and contrastive learning framework for identifying personality characteristics.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Adaptive information fusion network for multi‐modal personality recognition

Bao,

Liu,

et al. 2024

Computer Animation & Virtual

View full text Add to dashboard Cite

Personality recognition is of great significance in deepening the understanding of social relations. While personality recognition methods have made significant strides in recent years, the challenge of heterogeneity between modalities during feature fusion still needs to be solved. This paper introduces an adaptive multi‐modal information fusion network (AMIF‐Net) capable of concurrently processing video, audio, and text data. First, utilizing the AMIF‐Net encoder, we process the extracted audio and video features separately, effectively capturing long‐term data relationships. Then, adding adaptive elements in the fusion network can alleviate the problem of heterogeneity between modes. Lastly, we concatenate audio‐video and text features into a regression network to obtain Big Five personality trait scores. Furthermore, we introduce a novel loss function to address the problem of training inaccuracies, taking advantage of its unique property of exhibiting a peak at the critical mean. Our tests on the ChaLearn First Impressions V2 multi‐modal dataset show partial performance surpassing state‐of‐the‐art networks.

show abstract

Section: Related Workmentioning

confidence: 99%

“…In recent years, deep learning has become a mainstream method in personality recognition research. Researchers mostly conduct research based on multi‐modal methods 4,5 . Different data modalities for analysis and modeling are used to achieve data understanding and information extraction.…”

Section: Introductionmentioning

confidence: 99%

Adaptive information fusion network for multi‐modal personality recognition

Bao,

Liu,

et al. 2024

Computer Animation & Virtual

View full text Add to dashboard Cite

show abstract

“…The Image-grounded Text Decoder is activated by the Language Modeling Loss (LM) objective function, and its goal is to generate textual descriptions of a given image. an emotion-guided multi-modal fusion and contrastive learning framework [33]. The goal is to extract more profound and distinct features from various modalities and align features from different modalities .…”

Section: Multi-modal Fusionmentioning

confidence: 99%

Granularity Based Inter and Intra-Modal Fusion Network for Sarcasm Detection

Shi,

Zhao,

Chen

2023

Preprint

View full text Add to dashboard Cite

Multi-modal sarcasm detection is a task that involves detecting and identifyingsarcasm using multiple modalities of information. The key aspect of this task lies in how to model intra and inter-modality incongruity. Existing multi-modal sarcasm detection methods often focus on incongruity between modalities while overlooking the potential of fully exploring the semantic information within each modality. We tackle the problem in this paper by designing the Granularity Based Inter and Intra-Modal Fusion Network (GIIFN). Our approach combines tradi-tional image algorithms with deep learning models to extract comprehensive andrich semantic information from images. By incorporating a pre-trained languagemodel, we leverage knowledge from large-scale textual data to enhance the anal-ysis of images. Furthermore, our feature interaction model can effectively fuse and interact features at different granularities, capturing fine details and contex-tual information in the images. Through extensive experimental validation, we demonstrate the outstanding performance of our approach in multimodal satiredetection tasks, surpassing existing methods and outperforming state-of-the-art results.

show abstract