Visual and tactile interfaces for bi-directional human robot communication

Barber, Daniel; Lackey, Stephanie; Reinerman-Jones, Lauren; Hudson, Irwin

doi:10.1117/12.2015956

Cited by 13 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, the statistical classifier from Barber et al 18 Gestures fist markers and classified them. This statistical classifier was trained on 120 samples for each gesture from four researchers (30 samples each) collected prior to experimental data collection.…”

Section: Gesture Recognitionmentioning

confidence: 99%

“…To record and classify gesture commands from participants, a wireless IMU gesture glove used in previous experiments at UCF was used 18 The robot used for performing ISR missions was the Joint Reconnaissance and Mobility Robot (JRMBot). The JRMBot, Figure 4, is a small mobile platform containing various sensors for autonomous navigation including a Microsoft Kinect camera, laser range finder, and simulated compass/GPS via a Polhemus G4 six-degree-of-freedom tracking system.…”

Section: Equipmentmentioning

confidence: 99%

“…An alternative approach to visual classification of gestures is through Inertial Measurement Unit (IMU) devices. Previous efforts by Barber et al 18 compared multiple IMU devices for classification of 21 unique gestures, including several from the U.S. Army Field Manual, resulting in classification accuracies of 98%. Although this finding shows promise, classification was not demonstrated in real-time and the classifier model included training samples from users.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Speech and gesture interfaces for squad-level human-robot teaming

Harris

Barber

2014

SPIE Proceedings

Self Cite

View full text Add to dashboard Cite

As the military increasingly adopts semi-autonomous unmanned systems for military operations, utilizing redundant and intuitive interfaces for communication between Soldiers and robots is vital to mission success. Currently, Soldiers use a common lexicon to verbally and visually communicate maneuvers between teammates. In order for robots to be seamlessly integrated within mixed-initiative teams, they must be able to understand this lexicon. Recent innovations in gaming platforms have led to advancements in speech and gesture recognition technologies, but the reliability of these technologies for enabling communication in human robot teaming is unclear. The purpose for the present study is to investigate the performance of Commercial-Off-The-Shelf (COTS) speech and gesture recognition tools in classifying a Squad Level Vocabulary (SLV) for a spatial navigation reconnaissance and surveillance task. The SLV for this study was based on findings from a survey conducted with Soldiers at Fort Benning, GA. The items of the survey focused on the communication between the Soldier and the robot, specifically in regards to verbally instructing them to execute reconnaissance and surveillance tasks. Resulting commands, identified from the survey, were then converted to equivalent arm and hand gestures, leveraging existing visual signals (e.g. U.S. Army Field Manual for Visual Signaling). A study was then run to test the ability of commercially available automated speech recognition technologies and a gesture recognition glove to classify these commands in a simulated intelligence, surveillance, and reconnaissance task. This paper presents classification accuracy of these devices for both speech and gesture modalities independently.

show abstract

Section: Gesture Recognitionmentioning

confidence: 99%

Section: Equipmentmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Speech and gesture interfaces for squad-level human-robot teaming

Harris

Barber

2014

SPIE Proceedings

Self Cite

View full text Add to dashboard Cite

show abstract

“…Alternatively, some attention has been paid to established languages such as military gestures [12,14]. Worn sensors like data gloves have become popular, including use for control of robotic vehicles [15], because they do not require holding a sensor in the hand. The closest work to SID is perhaps the WITAS project [16], which focused on multi-modal dialogue for UAV operations.…”

Section: Related Workmentioning

confidence: 99%

Natural interaction for unmanned systems

Taylor¹,

Purman²,

Schermerhorn³

et al. 2015

SPIE Proceedings

View full text Add to dashboard Cite

Military unmanned systems today are typically controlled by two methods: tele-operation or menu-based, search-andclick interfaces. Both approaches require the operator's constant vigilance: tele-operation requires constant input to drive the vehicle inch by inch; a menu-based interface requires eyes on the screen in order to search through alternatives and select the right menu item. In both cases, operators spend most of their time and attention driving and minding the unmanned systems rather than on being a warfighter. With these approaches, the platform and interface become more of a burden than a benefit. The availability of inexpensive sensor systems in products such as Microsoft Kinect™ or Nintendo Wii™ has resulted in new ways of interacting with computing systems, but new sensors alone are not enough. Developing useful and usable human-system interfaces requires understanding users and interaction in context: not just what new sensors afford in terms of interaction, but how users want to interact with these systems, for what purpose, and how sensors might enable those interactions. Additionally, the system needs to reliably make sense of the user's inputs in context, translate that interpretation into commands for the unmanned system, and give feedback to the user. In this paper, we describe an example natural interface for unmanned systems, called the Smart Interaction Device (SID), which enables natural two-way interaction with unmanned systems including the use of speech, sketch, and gestures. We present a few example applications SID to different types of unmanned systems and different kinds of interactions.

show abstract

“…The ability of these technologies to engage natural modes of interaction through which information is conveyed, such as gesture and speech, have enabled efficient human-robot teaming 1,2 . The notion of intuitive and natural communication with robots has been a motivating factor in a variety of domains, specifically military 3 .…”

Section: Introductionmentioning

confidence: 99%

Technological evaluation of gesture and speech interfaces for enabling dismounted soldier-robot dialogue

et al. 2016

View full text Add to dashboard Cite

With increasing necessity for intuitive Soldier-robot communication in military operations and advancements in interactive technologies, autonomous robots have transitioned from assistance tools to functional and operational teammates able to service an array of military operations. Despite improvements in gesture and speech recognition technologies, their effectiveness in supporting Soldier-robot communication is still uncertain. The purpose of the present study was to evaluate the performance of gesture and speech interface technologies to facilitate Soldier-robot communication during a spatialnavigation task with an autonomous robot. Gesture and speech semantically based spatial-navigation commands leveraged existing lexicons for visual and verbal communication from the U.S Army field manual for visual signaling and a previously established Squad Level Vocabulary (SLV). Speech commands were recorded by a Lapel microphone and Microsoft Kinect, and classified by commercial off-the-shelf automatic speech recognition (ASR) software. Visual signals were captured and classified using a custom wireless gesture glove and software. Participants in the experiment commanded a robot to complete a simulated ISR mission in a scaled down urban scenario by delivering a sequence of gesture and speech commands, both individually and simultaneously, to the robot. Performance and reliability of gesture and speech hardware interfaces and recognition tools were analyzed and reported. Analysis of experimental results demonstrated the employed gesture technology has significant potential for enabling bidirectional Soldier-robot team dialogue based on the high classification accuracy and minimal training required to perform gesture commands.

show abstract

Visual and tactile interfaces for bi-directional human robot communication

Cited by 13 publications

References 0 publications

Speech and gesture interfaces for squad-level human-robot teaming

Speech and gesture interfaces for squad-level human-robot teaming

Natural interaction for unmanned systems

Technological evaluation of gesture and speech interfaces for enabling dismounted soldier-robot dialogue

Contact Info

Product

Resources

About