MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences

Abstract

Human communication is multimodal in nature; it is through multiple modalities such as language, voice, and facial expressions, that opinions and emotions are expressed. Data in this domain exhibits complex multi-relational and temporal interactions. Learning from this data is a fundamentally challenging research problem. In this paper, we propose Modal-Temporal Attention Graph (MTAG). MTAG is an interpretable graph-based neural model that provides a suitable framework for analyzing multimodal sequential data. We first introduce a procedure to convert unaligned multimodal sequence data into a graph with heterogeneous nodes and edges that captures the rich interactions across modalities and through time. Then, a novel graph fusion operation, called MTAG fusion, along with a dynamic pruning and read-out technique, is designed to efficiently process this modal-temporal graph and capture various interactions. By learning to focus only on the important interactions within the graph, MTAG achieves state-ofthe-art performance on multimodal sentiment analysis and emotion recognition benchmarks, while utilizing significantly fewer model parameters. 1

Face-to-Face Contrastive Learning for Social Intelligence Question-Answering

“…[14] show that multimodal transformers can be considered a special case of fully connected GNN's, including large multimodal transformers such as MERLOT [41] and UNITER [6]. Most similar to our work is [34], who use a graph neural network to model cross-modality, within-modality, and temporal connections, but differs from our strategy in our factorization strategy and contrastive learning setup. c) Self-Supervised Contrastive Learning: Selfsupervised learning attempts to create "labels" from data (e.g.…”

“…For a supervised training (baselines we will introduce next), we model the modality-modality components of each graph similar to the fashion in [34], and implemented in the Pytorch-Geometric package [10]. We add the following edges and associated graph convolutions to the graph: the edge connecting questions, the edge connecting answers, the edge connecting questions and answers, and the edge connecting factorization nodes z s to the question and answer nodes.…”

“…1) Self-Supervised Baselines: The self-supervised baselines we test perform the same graph augmentations and contrastive pre-training but with three different ways of creating factorization nodes: none (one video-level factorization node created by averaging all node representations as in [34]), and mean (segment the graph, create factorization nodes for each segment using mean readout of nodes in the subgraph).…”

Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension

“…(Zhang et al, 2021a) propose a multimodal graph fusion approach for named entity recognition, which conducted graph encoding via multimodal semantic interaction. (Yang et al, 2021) focus on multimodal sentiment analysis and emotion recognition, which unified video, audio, and text modalities into an attention graph and learned the interaction through graph fusion, dynamic pruning, and the read-out technique.…”

Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue System

“…Multimodal modeling is a research hotspot that has attracted the attention of many scholars in recent years [2,64]. The key to multimodal modeling includes multimodal fusion [53,56], consistency and difference [16,55], modality alignment [47]. We recommend the survey [2] for a comprehensive understanding.…”