Multimodal representation models for prediction and control from partial information

Abstract

Similar to humans, robots benefit from interacting with their environment through a number of different sensor modalities, such as vision, touch, sound. However, learning from different sensor modalities is difficult, because the learning model must be able to handle diverse types of signals, and learn a coherent representation even when parts of the sensor inputs are missing. In this paper, a multimodal variational autoencoder is proposed to enable an iCub humanoid robot to learn representations of its sensorimotor capabilities from different sensor modalities. The proposed model is able to (1) reconstruct missing sensory modalities, (2) predict the sensorimotor state of self and the visual trajectories of other agents actions, and (3) control the agent to imitate an observed visual trajectory. Also, the proposed multimodal variational autoencoder can capture the kinematic redundancy of the robot motion through the learned probability distribution. Training multimodal models is not trivial due to the combinatorial complexity given by the possibility of missing modalities. We propose a strategy to train multimodal models, which successfully achieves improved performance of different reconstruction models. Finally, extensive experiments have been carried out using an iCub humanoid robot, showing high performance in multiple reconstruction, prediction and imitation tasks.

Multi-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learning

“…6). 1 Given a single modality of clean image data, Dreamer is generally able to achieve high rewards on the tasks tested [5], [9].…”

“…Specifically, MuMMI uses PoE fusion [18], which was previously used in a multi-modal variational autoencoder [3] that was later extended to sequential settings [4]. Multi-modal models have also been adopted in robotics applications, where feature vectors from different modalities are concatenated into a single latent representation [1], [2]. Lately, PoE-based fusion has been applied to multi-modal self-supervised training [8], but unlike MuMMI, the method relies on hand-crafted taskdependent losses.…”

“…Dreamer uses a reconstruction-based ELBO, whilst CVRL is trained using a contrastive loss (but without a product-of-experts fusion layer). For both models, feature vectors extracted from modality-specific deep networks are fused via concatenation and missing observations are masked with zeros (similar to prior work [4], [1]). We also tested MuMMI-b; a variant of MuMMI with a modified density ratio estimator:…”

“…We observed that MuMMI degrades gracefully with greater amount of missing data, but remains robust compared to the other methods. Between the concatenation fusion methods (Dreamer and CVRL), Dreamer has poorer performance, despite given access to the clean depth images 1 . In comparison, CVRL was better able to learn from multiple modalities; we posit that Dreamer reconstruction loss does not permit the model to neglect the irrelevant inputs, which hampered learning of a good latent code.…”

“…and planning. Compared to deep models that "concatenate" different modalities [1], [2], structural modularity in our probabilistic graphical model provides a principled technique for dealing with missing data (rather than masking) with fewer parameters.…”