CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality

Abstract

Previous studies in multimodal sentiment analysis have used limited datasets, which only contain unified multimodal annotations. However, the unified annotations do not always reflect the independent sentiment of single modalities and limit the model to capture the difference between modalities. In this paper, we introduce a Chinese single-and multimodal sentiment analysis dataset, CH-SIMS, which contains 2,281 refined video segments in the wild with both multimodal and independent unimodal annotations. It allows researchers to study the interaction between modalities or use independent unimodal annotations for unimodal sentiment analysis. Furthermore, we propose a multi-task learning framework based on late fusion as the baseline. Extensive experiments on the CH-SIMS show that our methods achieve state-of-the-art performance and learn more distinctive unimodal representations. The full dataset and codes are available for use at https://github.com/ thuiar/MMSA.

Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis

“…In this work, experiments are conducted on two public multimodal sentiment analysis datasets, MOSI [31] and SIMS [28]. The basic statistics of each dataset are shown in Table 1.…”

“…SIMS. The SIMS dataset [28] is a Chinese MSA benchmark dataset with fine-grained uni-modal annotations. The dataset comprises of 2,281 refined video clips collected from different movies, TV serials, and variety shows with spontaneous expressions, various head poses, occlusions, and illuminations.…”

Neural Dialogue Generation Methods in Open Domain: A Survey

“…10 CH-SIMS The dataset contains 2,281 refined video segments in the wild with both multi-modal and independent unimodal annotations. 2020 [74] Informativeness and Consistency are usually evaluated by human evaluation.…”

CMU-MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French

“…It contains over 23, 000 sentences from across 1000 speakers and 250 topics. CH-SIMS (Yu et al, 2020) is a dataset of Chinese multimodal sentiment analysis with fine-grained annotations of sentiment per modality. IEMOCAP (Busso et al, 2008) is an inlab recorded dataset which consists of 151 videos of scripted dialogues between acting participants.…”

“…On a daily basis across the world, intentions and emotions are conveyed through joint utilization of these three modalities. While English, Chinese, and Spanish languages have resources for computational analysis of multimodal language (focusing on analysis of sentiment, subjectivity, or emotions (Yu et al, 2020;Zadeh et al, 2018b;Park et al, 2014;Wöllmer et al, 2013;Poria et al, 2020)), other commonly spoken languages across the globe lag behind. As Artificial Intelligence (AI) increasingly blends into everyday life across the globe, there is a genuine need for intelligent entities capable of understanding multimodal language in different cultures.…”