Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization

Liang, Paul Pu; Liu, Zhun; Tsai, Yao-Hung Hubert; Zhao, Qibin; Salakhutdinov, Ruslan; Morency, Louis–Philippe

doi:10.18653/v1/p19-1152

Cited by 49 publications

(11 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There exist various exciting recent work on improved multimodal fusion techniques Liang et al, 2019a;Pham et al, 2019;Baltrušaitis et al, 2019). In addition to the simplified feature and modality concatenations, we plan to explore some of these promising tensor-based multimodal fusion networks (Liu et al, 2018;Liang et al, 2019b;Tsai et al, 2019) for more robust intent classification on AMIE dataset as future work.…”

Section: Discussionmentioning

confidence: 99%

Audio-Visual Understanding of Passenger Intents for In-Cabin Conversational Agents

Okur¹,

Kumar²,

Sahay³

et al. 2020

Second Grand-Challenge and Workshop on Multimodal Language (Challenge-Hml)

View full text Add to dashboard Cite

Building multimodal dialogue understanding capabilities situated in the in-cabin context is crucial to enhance passenger comfort in autonomous vehicle (AV) interaction systems. To this end, understanding passenger intents from spoken interactions and vehicle vision systems is an important building block for developing contextual and visually grounded conversational agents for AV. Towards this goal, we explore AMIE (Automated-vehicle Multimodal In-cabin Experience), the in-cabin agent responsible for handling multimodal passenger-vehicle interactions. In this work, we discuss the benefits of multimodal understanding of in-cabin utterances by incorporating verbal/language input together with the non-verbal/acoustic and visual input from inside and outside the vehicle. Our experimental results outperformed text-only baselines as we achieved improved performances for intent detection with multimodal approach.

show abstract

Section: Discussionmentioning

confidence: 99%

Audio-Visual Understanding of Passenger Intents for In-Cabin Conversational Agents

Okur¹,

Kumar²,

Sahay³

et al. 2020

Second Grand-Challenge and Workshop on Multimodal Language (Challenge-Hml)

View full text Add to dashboard Cite

show abstract

“…Low-rank imputation: Complete multimodal data exhibits correlations between different modalities and leads to the low-rank data matrix. However, incomplete data breaks these correlations and increases tensor rank [19], [20]. To capture multimodal correlations, previous works project data into a common space by using low-rankness.…”

Section: Imputation Methodsmentioning

confidence: 99%

“…Besides nuclear norm, Fan et al [23] also minimized tensor tubal rank to deal with various missing patterns. Furthermore, Liang et al [20] combined the strength of non-linear functions to learn complex correlations in tensor rank minimization. However, these methods are usually computationally expensive for big data [24].…”

Section: Imputation Methodsmentioning

confidence: 99%

GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation

Zheng¹,

Chen²,

Sun³

et al. 2022

Preprint

View full text Add to dashboard Cite

Conversations have become a critical data format on social media platforms. Understanding conversation from emotion, content, and other aspects also attracts increasing attention from researchers due to its widespread application in human-computer interaction. In real-world environments, we often encounter the problem of incomplete modalities, which has become a core issue of conversation understanding. To address this problem, researchers propose various methods. However, existing approaches are mainly designed for individual utterances or medical images rather than conversational data, which cannot exploit temporal and speaker information in conversations. To this end, we propose a novel framework for incomplete multimodal learning in conversations, called "Graph Complete Network (GCNet)", filling the gap of existing works. Our GCNet contains two well-designed graph neural network-based modules, "Speaker GNN" and "Temporal GNN", to capture temporal and speaker information in conversations. To make full use of complete and incomplete data in feature learning, we jointly optimize classification and reconstruction in an end-to-end manner. To verify the effectiveness of our method, we conduct experiments on three benchmark conversational datasets. Experimental results demonstrate that our GCNet is superior to existing state-of-the-art approaches in incomplete multimodal learning.

show abstract

“…The optimization objectives for the core tensor are presented in Equs. (8) and (9). The optimization objectives for the factor matrix are presented in Equ.…”

Section: Stochastic Optimizationmentioning

confidence: 99%

“…Here, during the learning process, each modality corresponds to a feature and the feature alignment involves fusion. Tensors are a common form of feature fusion for multi-modal learning [7], [8], [9], [10]. Unfortunately, tensors can be difficult to process in practice.…”

mentioning

confidence: 99%

SGD_Tucker: A Novel Stochastic Optimization Strategy for Scalable Parallel Sparse Tucker Decomposition

Rellermeyer

et al. 2021

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Sparse Tucker Decomposition (STD) algorithms learn a core tensor and a group of factor matrices to obtain an optimal low-rank representation feature for the High-Order, High-Dimension, and Sparse Tensor (HOHDST). However, existing STD algorithms face the problem of intermediate variables explosion which results from the fact that the formation of those variables, i.e., matrices Khatri-Rao product, Kronecker product, and matrix-matrix multiplication, follows the whole elements in sparse tensor. The above problems prevent deep fusion of efficient computation and big data platforms. To overcome the bottleneck, a novel stochastic optimization strategy (SGD Tucker) is proposed for STD which can automatically divide the high-dimension intermediate variables into small batches of intermediate matrices. Specifically, SGD Tucker only follows the randomly selected small samples rather than the whole elements, while maintaining the overall accuracy and convergence rate. In practice, SGD Tucker features the two distinct advancements over the state of the art. First, SGD Tucker can prune the communication overhead for the core tensor in distributed settings. Second, the low data-dependence of SGD Tucker enables fine-grained parallelization, which makes SGD Tucker obtaining lower computational overheads with the same accuracy. Experimental results show that SGD Tucker runs at least 2X faster than the state of the art.

show abstract

Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization

Cited by 49 publications

References 44 publications

Audio-Visual Understanding of Passenger Intents for In-Cabin Conversational Agents

Audio-Visual Understanding of Passenger Intents for In-Cabin Conversational Agents

GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation

SGD_Tucker: A Novel Stochastic Optimization Strategy for Scalable Parallel Sparse Tucker Decomposition

Contact Info

Product

Resources

About