Object Permanence Through Audio-Visual Representations

Abstract

As robots perform manipulation tasks and interact with objects, it is probable that they accidentally drop objects that subsequently bounce out of their visual fields (e.g., due to an inadequate grasp of an unfamiliar object). To enable robots to recover from such errors, we draw upon the concept of object permanence-objects remain in existence even when they are not being sensed (e.g., seen) directly. In particular, we developed a multimodal neural network model-using a partial, observed bounce trajectory and the audio resulting from drop impact as its inputs-to predict the full bounce trajectory and the end location of a dropped object. We empirically show that: (1) our multimodal method predicted end locations close in proximity (i.e., within the visual field of the robot's wrist camera) to the actual locations and (2) the robot was able to retrieve dropped objects by applying minimal vision-based pick-up adjustments. Additionally, we show that our multimodal method outperformed the vision-only and audio-only baselines in retrieving dropped objects. Our results provide insights in enabling object permanence for robots and offer foundations for ensuring robust robot autonomy in task execution.

The Boombox: Visual Reconstruction from Acoustic Vibrations

“…There are works that, given vision, enhance sounds [30,18], fill in missing sounds [42], and generate sounds entirely from video [32,43]. Further, there have been recent works in integrating vision and sound to improve recognition of environmental properties [3,21,8] and object properties, such as geometry and materials [40,39]. Lastly, there have been works in using audiovisual data for representation learning [33,4,28].…”