A Human-Centered Approach to One-Shot Gesture Learning

Abstract

This article discusses the problem of one-shot gesture recognition using a humancentered approach and its potential application to fields such as human-robot interaction where the user's intentions are indicated through spontaneous gesturing (one shot). Casual users have limited time to learn the gestures interface, which makes one-shot recognition an attractive alternative to interface customization. In the aim of natural interaction with machines, a framework must be developed to include the ability of humans to understand gestures from a single observation. Previous approaches to oneshot gesture recognition have relied heavily on statistical and data-mining-based solutions and have ignored the mechanisms that are used by humans to perceive and execute gestures and that can provide valuable context information. This omission has led to suboptimal solutions. The focus of this study is on the process that leads to the realization of a gesture, rather than on the gesture itself. In this case, context involves the way in which humans produce gestures-the kinematic and anthropometric characteristics. In the method presented here, the strategy is to generate a data set of realistic samples based on features extracted from a single gesture sample. These features, called the "gist of a gesture," are considered to represent what humans remember when seeing a gesture and, later, the cognitive process involved when trying to replicate it. By adding meaningful variability to these features, a large training data set is created while preserving the fundamental structure of the original gesture. The availability of a large data set of realistic samples allows the use of training classifiers for future recognition. The performance of the method is evaluated using different lexicons, and its efficiency is compared with that of traditional N-shot learning approaches. The strength of the approach is further illustrated through human and machine recognition of gestures performed by a dual-arm robotic platform.

Learning Semantics of Gestural Instructions for Human-Robot Collaboration

“…One possibility could be to incorporate on-the-fly gesture learning in the PIL framework. It is a topic of active research in gesture recognition (Fanello et al, 2013 ; Cabrera and Wachs, 2017 ; Cabrera et al, 2017 ). Once a gesture is recorded, the user can teach its association with a robot action.…”

O2A: One-Shot Observational Learning with Action Vectors

“…Most of the early methods required assistance in observing the demonstrations. This assistance was provided by motion capture systems (Ijspeert et al, 2001;Ijspeert et al, 2002;Field et al, 2009), visual detectors (Ramirez-Amaro et al, 2017;Sieb et al, 2020;Zhang and Nikolaidis, 2019), skeleton tracking (Cabrera and Wachs, 2017), trackers/markers (Dillmann, 2004;Dragan and Srinivasa, 2012;Gupta et al, 2016) or a combination of the above (Kuniyoshi et al, 1994). However, the entities to be tracked or detected must be known beforehand and only demonstrations using these entities can be learned.…”

Introducing the NEMO-Lowlands iconic gesture dataset, collected through a gameful human–robot interaction

“…In order to distinguish between important movements and noise, and to also correct for differences in location due to the position or height of the participant, a pre-processing step is performed to identify salient features of the gestures, also known as primitives (Ramey et al, 2012 ). We based our approach on the work by Cabrera and Wachs ( 2017 ) by using the inflection points of the hands’ motion trajectories, combined with peaks in the hands’ position (Fig. 6 shows a time series trajectory where inflection points and peaks are marked).…”

“…For example, we calculate whether the hand was in front of or behind, and above or below the shoulder. Cabrera and Wachs ( 2017 ) call the resulting sequence of inflection points and relative locations the gist of the gesture. One limitation that remains is that the same gesture could be performed at different positions relative to the body.…”