Communication between robots and humans is a challenge in complex built environments. In this project, we are exploring how previous achievements in this area apply to human/robot communication within immersive virtual displays. In our concrete example, we host human-scale robots that interact with humans in our Collaborative-Research Augmented Immersive Virtual Environment Laboratory (CRAIVE-Lab). The CRAIVE-Lab provides a physical/digital environment for collaborative tasks using seamless video from multiple projectors and a 128-channel wave-field system. The system is designed to monitor the whole floor-space area (10 m × 12 m) with a camera network mounted to the ceiling grid and a microphone array combining shotgun, spherical and possibly people-worn microphones. Based on the sensor data, a visual analysis and an auditory scene analysis are performed. The latter includes sound localization, speech and musical-feature recognition. The robots, a Rethink Baxter robot mounted on an electric wheelchair and a Robokind Zeno, are used to perform assistive technology and social communication tasks. Both robots have direct access to these data using a digital feed over a wireless network to augment their own sensor systems of built-in cameras and microphones. [Work supported by NSF grant #1229391.]
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.