The Multimodal Sentiment Analysis in Car Reviews (MuSe-CaR) Dataset: Collection, Insights and Improvements

Abstract

Truly real-life data presents a strong, but exciting challenge for sentiment and emotion research. The high variety of possible 'in-the-wild' properties makes large datasets such as these indispensable with respect to building robust machine learning models. A sufficient quantity of data covering a deep variety in the challenges of each modality to force the exploratory analysis of the interplay of all modalities has not yet been made available in this context. In this contribution, we present MuSe-CaR, a first of its kind multimodal dataset. The data is publicly available as it recently served as the testing bed for the 1st Multimodal Sentiment Analysis Challenge, and focused on the tasks of emotion, emotion-target engagement, and trustworthiness recognition by means of comprehensively integrating the audio-visual and language modalities. Furthermore, we give a thorough overview of the dataset in terms of collection and annotation, including annotation tiers not used in this year's MuSe 2020. In addition, for one of the sub-challenges -predicting the level of trustworthiness -no participant outperformed the baseline model, and so we propose a simple, but highly efficient Multi-Head-Attention network that exceeds using multimodal fusion the baseline by around 0.2 CCC (almost 50 % improvement).

Sequential Late Fusion Technique for Multi-modal Sentiment Analysis

“…. $15.00 https://doi.org/10.1145/3453892.3461009 fusion (LF) [3,9] approaches are the most prominent techniques. As discussed in [4] by N.Majumder et al, sequential (or hierarchical in their case) late fusion can filter out inter-modal correlation.…”

GraphTMT: Unsupervised Graph-based Topic Modeling from Video Transcripts

“…The MuSe-CaR [30] is a multimodal data set gathered in-the-wild from English YouTube videos centred around car reviews. It was created with different computational tasks in mind, allowing researchers to improve the machine's understanding of how sentiment and topics are connected.…”

“…These pipelines provide timestamps for each word (start and end point of an utterance) through which all words articulated within a segment can be assigned to it. Due to the in-the-wild factors, the error rate of the automatic transcriptions is estimated to be relatively high and specified at around 25 % on a subset of 10 hand-transcribed videos by the authors of [30].…”

“…Compared to the ground-truth labels, the resulting topics are more fine-grained. For example, K-Means can distinguish between YouTube and Storage, both mapped to General Information [30]. As we increase k, the model infers more miscellaneous topics which decreases the coherence performance.…”

An Estimation of Online Video User Engagement From Features of Time- and Value-Continuous, Dimensional Emotions

“…Furthermore, we utilize a richly annotated data set of ca. 600 h of continuous annotations (Stappen et al, 2021), and derive cross-task features from this initial correlation analysis. Second, we compare these engineered, lean features, to a computationally intensive feature selection approach and to all features when predicting selected engagement indicators (i.e., views, likes, number of comments, likes of the comments).…”