A Multimodal Robot Based Model for the Preservation of Intangible Cultural Heritage

Montalvo, Melissa; Calle-Ortiz, Eduardo; Chica, Juan Beltrán

doi:10.1145/3029798.3038315

Cited by 5 publications

(1 citation statement)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, learning repre-* authors contributed equally sentations for these spoken utterances is a complex research problem due to the presence of multiple heterogeneous sources of information (Baltrušaitis et al, 2017). This challenging yet crucial research area has real-world applications in robotics (Montalvo et al, 2017;Noda et al, 2014), dialogue systems (Johnston et al, 2002;Rudnicky, 2005), intelligent tutoring systems (Mao and Li, 2012;Banda and Robinson, 2011;Pham and Wang, 2018), and healthcare diagnosis (Wentzel and van der Geest, 2016;Lisetti et al, 2003;Sonntag, 2017). Recent progress on multimodal representation learning has investigated various neural models that utilize one or more of attention, memory and recurrent components (Yang et al, 2017;.…”

Section: Introductionmentioning

confidence: 99%

Strong and Simple Baselines for Multimodal Utterance Embeddings

Liang

Lim

Tsai

et al. 2019

Proceedings of the 2019 Conference of the North

View full text Add to dashboard Cite

Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intonations. Learning representations for these spoken utterances is a complex research problem due to the presence of multiple heterogeneous sources of information. Recent advances in multimodal learning have followed the general trend of building more complex models that utilize various attention, memory and recurrent components. In this paper, we propose two simple but strong baselines to learn embeddings of multimodal utterances. The first baseline assumes a conditional factorization of the utterance into unimodal factors. Each unimodal factor is modeled using the simple form of a likelihood function obtained via a linear transformation of the embedding. We show that the optimal embedding can be derived in closed form by taking a weighted average of the unimodal features. In order to capture richer representations, our second baseline extends the first by factorizing into unimodal, bimodal, and trimodal factors, while retaining simplicity and efficiency during learning and inference. From a set of experiments across two tasks, we show strong performance on both supervised and semi-supervised multimodal prediction, as well as significant (10 times) speedups over neural models during inference. Overall, we believe that our strong baseline models offer new benchmarking options for future research in multimodal learning.

show abstract