A Framework for Sensorimotor Cross-Perception and Cross-Behavior Knowledge Transfer for Object Categorization

Abstract

From an early age, humans learn to develop an intuition for the physical nature of the objects around them by using exploratory behaviors. Such exploration provides observations of how objects feel, sound, look, and move as a result of actions applied on them. Previous works in robotics have shown that robots can also use such behaviors (e.g., lifting, pressing, shaking) to infer object properties that camera input alone cannot detect. Such learned representations are specific to each individual robot and cannot currently be transferred directly to another robot with different sensors and actions. Moreover, sensor failure can cause a robot to lose a specific sensory modality which may prevent it from using perceptual models that require it as input. To address these limitations, we propose a framework for knowledge transfer across behaviors and sensory modalities such that: (1) knowledge can be transferred from one or more robots to another, and, (2) knowledge can be transferred from one or more sensory modalities to another. We propose two different models for transfer based on variational auto-encoders and encoder-decoder networks. The main hypothesis behind our approach is that if two or more robots share multi-sensory object observations of a shared set of objects, then those observations can be used to establish mappings between multiple features spaces, each corresponding to a combination of an exploratory behavior and a sensory modality. We evaluate our approach on a category recognition task using a dataset in which a robot used 9 behaviors, coupled with 4 sensory modalities, performed multiple times on 100 objects. The results indicate that sensorimotor knowledge about objects can be transferred both across behaviors and across sensory modalities, such that a new robot (or the same robot, but with a different set of sensors) can bootstrap its category recognition models without having to exhaustively explore the full set of objects.

A Framework for Multisensory Foresight for Embodied Agents

“…Dataset. The dataset described in [37] is used to evaluate and compare the proposed network with the single-modal network. For collecting the dataset, an uppertorso humanoid robot with a 7-DOF arm manipulates 100 objects by executing 9 different exploratory behaviors (push, poke, press, shake, lift, drop, grasp, tap and hold) multiple times and records visual, haptic, auditory and vibrotactile sensory data.…”

“…Since different robots have different morphologies and different sensor suites, the learned knowledge cannot be directly used by another robot. An interesting avenue for future work is to extend transfer learning methodologies (e.g., [38], [37]) as to enable a robot to bootstrap its sensorimotor learning process with knowledge learned by another robot. Another viable direction for future work is to integrate the multisensory next-frame prediction methodology described here with reinforcement learning methods for object manipulation tasks.…”

Planning Multimodal Exploratory Actions for Online Robot Attribute Learning

“…One limitation of our existing framework is that the attribute recognition models are learned by a single robot and cannot directly be used by another robot that has different behaviors, morphology, and sensory modalities. We plan to use sensorimotor transfer learning (e.g., [37,38]) to scale up our framework to allow multiple different robots to learn such models and share their knowledge as to further speed up learning. In addition, considering correlations between attributes and handling fuzzy attributes can potentially improve the performance of On-RAL.…”

Planning Multimodal Exploratory Actions for Online Robot Attribute Learning

“…One limitation of our existing framework is that the attribute recognition models are learned by a single robot and cannot directly be used by another robot that has different behaviors, morphology, and sensory modalities. We plan to use sensorimotor transfer learning (e.g., [38,37]) to scale up our framework to allow multiple different robots to learn such models and share their knowledge as to further speed up learning. In addition, considering correlations between attributes and handling fuzzy attributes can potentially improve the performance of On-RAL.…”