Speech and gesture interfaces for squad-level human-robot teaming

Abstract

As the military increasingly adopts semi-autonomous unmanned systems for military operations, utilizing redundant and intuitive interfaces for communication between Soldiers and robots is vital to mission success. Currently, Soldiers use a common lexicon to verbally and visually communicate maneuvers between teammates. In order for robots to be seamlessly integrated within mixed-initiative teams, they must be able to understand this lexicon. Recent innovations in gaming platforms have led to advancements in speech and gesture recognition technologies, but the reliability of these technologies for enabling communication in human robot teaming is unclear. The purpose for the present study is to investigate the performance of Commercial-Off-The-Shelf (COTS) speech and gesture recognition tools in classifying a Squad Level Vocabulary (SLV) for a spatial navigation reconnaissance and surveillance task. The SLV for this study was based on findings from a survey conducted with Soldiers at Fort Benning, GA. The items of the survey focused on the communication between the Soldier and the robot, specifically in regards to verbally instructing them to execute reconnaissance and surveillance tasks. Resulting commands, identified from the survey, were then converted to equivalent arm and hand gestures, leveraging existing visual signals (e.g. U.S. Army Field Manual for Visual Signaling). A study was then run to test the ability of commercially available automated speech recognition technologies and a gesture recognition glove to classify these commands in a simulated intelligence, surveillance, and reconnaissance task. This paper presents classification accuracy of these devices for both speech and gesture modalities independently.

Achieving the Vision of Effective Soldier-Robot Teaming

“…This effort was performed in collaboration with our RCTA partners at the University of Central Florida and MIT (see [8][9][10] for related publications). A multimodal user interface (MMI; Figure 1) was developed to integrate several research products into a usable means of bi-directional communication with a robot.…”

Technological evaluation of gesture and speech interfaces for enabling dismounted soldier-robot dialogue

“…with closed fist). 6 The West Modified version of American Sign Language (ASL) 34 "West" gesture, but without the hand signal (eg. with closed fist).…”

“…Multimodal communication through interfaces that support natural language modalities such as gesture and speech have been demonstrated to be robust interaction methods for enabling bidirectional human-robot communication 1,6,7 . The integration of these interfaces for HRI plays a crucial role in facilitating the development of robust and autonomous robotic systems by supporting alternative interaction methods 1 .…”

“…A pilot study was conducted and reported on the ability of gesture and speech interfaces for classification of high level commands in a simulated ISR mission 6 . It focused mainly on classification of the high level commands using different speech recognition hardware and software, and illustrated the potential of gesture recognition for classifying those commands.…”

Effects of Varying Noise Levels and Lighting Levels on Multimodal Speech and Visual Gesture Interaction with Aerobots

“…Their project was aimed at studying, implementing, and validating NUIs efficiency in human UAV interaction [15]. Harris and Barber [16,17] investigated the performance of a speech and gesture multimodal interface for a soldier-robot team communication during an ISR mission. They also suggested the possibility of developing complex semantic navigation commands such as "perch over there (speech + pointing gesture), on the tank to the right of the stone monument (speech)" [16,18].…”