Low-Level Fusion of Audio and Video Feature for Multi-Modal Emotion Recognition

Abstract

Bimodal emotion recognition through audiovisual feature fusion has been shown superior over each individual modality in the past. Still, synchronization of the two streams is a challenge, as many vision approaches work on a frame basis opposing audio turn-or chunk-basis. Therefore, late fusion schemes such as simple logic or voting strategies are commonly used for the overall estimation of underlying affect. However, early fusion is known to be more effective in many other multimodal recognition tasks. We therefore suggest a combined analysis by descriptive statistics of audio and video Low-Level-Descriptors for subsequent static SVM Classification. This strategy also allows for a combined feature-space optimization which will be discussed herein. The high effectiveness of this approach is shown on a database of 11.5h containing six emotional situations in an airplane scenario.

DM2S2: Deep Multimodal Sequence Sets With Hierarchical Modality Attention

“…In general, DNN-based multimodal models include multiple streams of networks for modalities [14]. The models often have a component for fusing the features [2], [6], [15] to make a prediction based on intermediate features from the streams. Hence, by considering the fusion stage in the model architecture, multimodal architectures can be categorized into early fusion [15], mid-fusion [6], [7], [14], [16], and late fusion [2], [17].…”

“…The early fusion approach integrates the features of different modalities as inputs and uses a unified feature for the downstream task [15]. The mid-fusion approach involves the concatenation of features that are encoded from the raw data of each modality into a single feature [6], [7], [14], [16].…”

“…This encoder outputs a hidden state with dimension d = 1024. In the Production Ad-LP dataset, we used a pre-trained Japanese BERT 15 as the encoder. This encoder outputs a hidden state with dimension d = 768.…”