Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations

Abstract

Can conversational videos captured from multiple egocentric viewpoints reveal the map of a scene in a cost-efficient way? We seek to answer this question by proposing a new problem: efficiently building the map of a previously unseen 3D environment by exploiting shared information in the egocentric audio-visual observations of participants in a natural conversation. Our hypothesis is that as multiple people ("egos") move in a scene and talk among themselves, they receive rich audio-visual cues that can help uncover the unseen areas of the scene. Given the high cost of continuously processing egocentric visual streams, we further explore how to actively coordinate the sampling of visual information, so as to minimize redundancy and reduce power use. To that end, we present an audio-visual deep reinforcement learning approach that works with our shared scene mapper to selectively turn on the camera to efficiently chart out the space. We evaluate the approach using a state-of-the-art audio-visual simulator for 3D scenes as well as real-world video. Our model outperforms previous state-of-the-art mapping methods, and achieves an excellent cost-accuracy tradeoff. Project: http://vision.cs. utexas.edu/projects/chat2map.

Move2Hear: Active Audio-Visual Source Separation

“…Audio-visual spatial correspondence learning Learning the spatial alignment between video and audio is important for self-supervision [77,50,75,66], spatial audio generation [51,21,63,7,45], audio-visual embodied learning [8,44,46,9] and 3D scene mapping [62,47]. However, these methods are either restricted to exocentric settings [51,77,50,21,63,66,7], or else tackle egocentric settings [46,45,9,47] in simulated 3D environments that lack realism and diversity, both in terms of the audio-visual content of the videos and the continuous camera motion due to the camera-wearer's physical movements. On the contrary, we learn an audio-visual representation from real-world egocentric video.…”