See, feel, act: Hierarchical learning for complex manipulation skills with multisensory fusion

Abstract

Humans are able to seamlessly integrate tactile and visual stimuli with their intuitions to explore and execute complex manipulation skills. They not only see but also feel their actions. Most current robotic learning methodologies exploit recent progress in computer vision and deep learning to acquire data-hungry pixel-to-action policies. These methodologies do not exploit intuitive latent structure in physics or tactile signatures. Tactile reasoning is omnipresent in the animal kingdom, yet it is underdeveloped in robotic manipulation. Tactile stimuli are only acquired through invasive interaction, and interpretation of the data stream together with visual stimuli is challenging. Here, we propose a methodology to emulate hierarchical reasoning and multisensory fusion in a robot that learns to play Jenga, a complex game that requires physical interaction to be played effectively. The game mechanics were formulated as a generative process using a temporal hierarchical Bayesian model, with representations for both behavioral archetypes and noisy block states. This model captured descriptive latent structures, and the robot learned probabilistic models of these relationships in force and visual domains through a short exploration phase. Once learned, the robot used this representation to infer block behavior patterns and states as it played the game. Using its inferred beliefs, the robot adjusted its behavior with respect to both its current actions and its game strategy, similar to the way humans play the game. We evaluated the performance of the approach against three standard baselines and show its fidelity on a real-world implementation of the game.

Using Logic to Evolve More Logic: Composing Logical Operators via Self-Assembly

“…Such compositional processes (more generally) might be instantiated in nature in terms of the computational principles or neurobiological underpinnings of any adaptive decision-making process. Modular composition may arise, and aid efficacy or efficiency of such processes, in several different settings; for example, composing multiple sensory modalities, such as tactile and visual stimulation (Fazeli et al, 2019); arranging dominance-relations to form a hierarchical representation of a social group (Seyfarth and Cheney, 2018); cognitive reasoning involving hierarchically organised decision-making (Sarafyazd and Jazayeri, 2019); or other such functionaldemand protocols in nature, such as the availability of food, density of populations, and presence of predators in migratory species (Hopcraft et al, 2014).…”

Manipulation Skill Acquisition for Robotic Assembly Based on Multi-Modal Information Description

“…Reinforcement learning has been previously applied to manipulation learning [21]- [24]. Recently, much research has focused on using deep reinforcement learning to make robots play building block games [25], [26]and significant advances were made. In addition, significant amount of work has been done on robot grasbing [27], [28], door opening [29], navigating [30].…”

Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks

“…Yet few algorithms endow robots with a similar capability. While the utility of multimodal data has frequently been shown in robotics [7,20,54,58,66], the proposed manipulation strategies often rely on handcrafted features or prior knowledge about how to solve a task. This makes many of these methods task-specific.…”