Recognizing Emotions in Video Using Multimodal DNN Feature Fusion

Williams, Jennifer; Kleinegesse, Steven; Comanescu, Ramona; Radu, Oana

doi:10.18653/v1/w18-3302

Cited by 60 publications

(15 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MSA aims to predict the people's sentiment from the video, audio, and text of the utterances. The models like MFN [30] and EF-LSTM [27] can work on aligned multimodal data, which means the frames of audio and vision have explicit correspondence with the words in the text modality. To deal with more practical scenarios, MSA models are gradually expanding to the area of unaligned multimodal data inputs.…”

Section: Multimodal Sentiment Analysismentioning

confidence: 99%

Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis

Yuan

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Improving robustness against data missing has become one of the core challenges in Multimodal Sentiment Analysis (MSA), which aims to judge speaker sentiments from the language, visual, and acoustic signals. In the current research, translation-based methods and tensor regularization methods are proposed for MSA with incomplete modality features. However, both of them fail to cope with random modality feature missing in non-aligned sequences. In this paper, a transformer-based feature reconstruction network (TFR-Net) is proposed to improve the robustness of models for the random missing in non-aligned modality sequences. First, intramodal and inter-modal attention-based extractors are adopted to learn robust representations for each element in modality sequences. Then, a reconstruction module is proposed to generate the missing modality features. With the supervision of SmoothL1Loss between generated and complete sequences, TFR-Net is expected to learn semantic-level features corresponding to missing features. Extensive experiments on two public benchmark datasets show that our model achieves good results against data missing across various missing modality combinations and various missing degrees. CCS CONCEPTS• Information systems → Multimedia and multimodal retrieval; Web log analysis; Web applications; • Computing methodologies → Natural language processing.

show abstract

Section: Multimodal Sentiment Analysismentioning

confidence: 99%

Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis

Yuan

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…This method is used in many models. In [23], the author directly performs input-level feature fusion on multimodal data in the input stage and combines deep neural networks for sentiment analysis. In literature [24,25], the author first encodes each modal separately and then uses feature-level fusion in the middle layer to obtain multimodal embedding, which is static feature-level fusion.…”

Section: Multimodal Sentiment Analysismentioning

confidence: 99%

Feature Extraction Network with Attention Mechanism for Data Enhancement and Recombination Fusion for Multimodal Sentiment Analysis

Lin

Zhang

2021

Information

View full text Add to dashboard Cite

Multimodal sentiment analysis and emotion recognition represent a major research direction in natural language processing (NLP). With the rapid development of online media, people often express their emotions on a topic in the form of video, and the signals it transmits are multimodal, including language, visual, and audio. Therefore, the traditional unimodal sentiment analysis method is no longer applicable, which requires the establishment of a fusion model of multimodal information to obtain sentiment understanding. In previous studies, scholars used the feature vector cascade method when fusing multimodal data at each time step in the middle layer. This method puts each modal information in the same position and does not distinguish between strong modal information and weak modal information among multiple modalities. At the same time, this method does not pay attention to the embedding characteristics of multimodal signals across the time dimension. In response to the above problems, this paper proposes a new method and model for processing multimodal signals, which takes into account the delay and hysteresis characteristics of multimodal signals across the time dimension. The purpose is to obtain a multimodal fusion feature emotion analysis representation. We evaluate our method on the multimodal sentiment analysis benchmark dataset CMU Multimodal Opinion Sentiment and Emotion Intensity Corpus (CMU-MOSEI). We compare our proposed method with the state-of-the-art model and show excellent results.

show abstract

“…It integrates the contextual information encoded in word embeddings space and the extracted glyph representation of sequential handwritten characters, containing two-modal features. Inspired by the previous fusion methods of multi-modal representation such as Early Fusion LSTM [35], Tensor Fusion Network [40], Memory Fusion Network [41] and Low-rank multi-modal Fusion [21], we design a multi-modal fusion sub-layer in each block of DSTFN to tackle the above features.…”

Section: Related Workmentioning

confidence: 99%

Sentence-level Online Handwritten Chinese Character Recognition

Li,

Yang,

Chen

et al. 2021

Preprint

View full text Add to dashboard Cite

Single online handwritten Chinese character recognition (single OLHCCR) has achieved prominent performance. However, in real application scenarios, users always write multiple Chinese characters to form one complete sentence and the contextual information within these characters holds the significant potential to improve the accuracy, robustness and efficiency of sentence-level OLHCCR. In this work, we first propose a simple and straightforward endto-end network, namely vanilla compositional network (VCN) to tackle the sentence-level OLHCCR. It couples convolutional neural network with sequence modeling architecture to exploit the handwritten character's previous contextual information. Although VCN performs much better than the state-of-the-art single OLHCCR model, it exposes high fragility when confronting with not well written characters such as sloppy writing, missing or broken strokes. To improve the robustness of sentence-level OLHCCR, we further propose a novel deep spatial-temporal fusion network (DSTFN). It utilizes a pre-trained autoregresssive framework as the backbone component, which projects each Chinese character into word embeddings, and integrates the spatial glyph features of handwritten characters and their contextual information multiple times at multi-layer fusion module. We also construct a large-scale sentencelevel handwriting dataset, named as CSOHD to evaluate models. Extensive experiment results demonstrate that DSTFN achieves the state-of-the-art performance, which presents strong robustness compared with VCN and exiting single OLHCCR models. The in-depth empirical analysis and case studies indicate that DSTFN can significantly improve the efficiency of handwriting input, with

show abstract

Recognizing Emotions in Video Using Multimodal DNN Feature Fusion

Cited by 60 publications

References 26 publications

Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis

Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis

Feature Extraction Network with Attention Mechanism for Data Enhancement and Recombination Fusion for Multimodal Sentiment Analysis

Sentence-level Online Handwritten Chinese Character Recognition

Contact Info

Product

Resources

About