Deep unsupervised network for multimodal perception, representation and classification

Abstract

In this paper, we tackle the problem of multimodal learning for autonomous robots. Autonomous robots interacting with humans in an evolving environment need the ability to acquire knowledge from their multiple perceptual channels in an unsupervised way. Most of the approaches in the literature exploit engineered methods to process each perceptual modality. In contrast, robots should be able to acquire their own features from the raw sensors, leveraging the information elicited by interaction with their environment: learning from their sensorimotor experience would result in a more efficient strategy in a life-long perspective. To this end, we propose an architecture based on deep networks, which is used by the humanoid robot iCub to learn a task from multiple perceptual modalities (proprioception, vision, audition). By structuring high-dimensional, multimodal information into a set of distinct sub-manifolds in a fully unsupervised way, it performs a substantial dimensionality reduction by providing both a symbolic representation of data and a fine discrimination between two similar stimuli. Moreover, the proposed network is able to exploit multimodal correlations to improve the representation of each modality alone.

Multimodal representation models for prediction and control from partial information

“…It is trained as (i), which is also the approach Figure 11: Visualization of performance scores (prediction error) of the proposed method and other methods. Our training strategy clearly improves performance of reconstruction methods, including Vanilla VAE and the model proposed in [50]. Our method performs equally or better than the alternatives in the complex fully sensorimotor state estimation task.…”

“…In future work, we will investigate how to leverage the generative capabilities of the network, and how this method can be combined with more advanced exploration strategies (such as curiosity-based strategies) in order to acquire a self-perception database that covers the robot and environment states as much as possible [55,56]. The presented method will also be combined with perspective taking mechanisms [9,10] The implementation of the comparison architecture from [50] is based on the source code provided by the authors and replicate most of its parameters. Only differences are the number of modalities (set to 5), the number of parameters (set to 100), and the number of classes (set to one as classification is not considered here).…”

“…(iii) The multimodal architecture proposed in [50]. This architecture learns a shared latent representation and classification of the inputs, and can be used to reconstruct missing modalities.…”

Seamless Integration and Coordination of Cognitive Skills in Humanoid Robots: A Deep Learning Approach

“…It is essential to link these skills with synergy by developing spatio-temporal coordination among them. Furthermore, these skills ideally arise from the robot's experience (of reaching for and grasping objects, for example), rather than from hand-engineered features reflecting a human engineer's understanding of what any given task may require [1][2][3][4][5][6][7][8][9].…”

Multi-Model Stability Control Method of Underactuated Biped Robots Based on Imbalance Degrees

“…Simulation results have shown that the algorithm has a wide stability range. It can make the robots rapidly restore upright posture after power-on or switching from walking to standing position, and keep the robots stable [12][13][14][15].…”