Using gaze patterns to predict task intent in collaboration

Abstract

In everyday interactions, humans naturally exhibit behavioral cues, such as gaze and head movements, that signal their intentions while interpreting the behavioral cues of others to predict their intentions. Such intention prediction enables each partner to adapt their behaviors to the intent of others, serving a critical role in joint action where parties work together to achieve a common goal. Among behavioral cues, eye gaze is particularly important in understanding a person's attention and intention. In this work, we seek to quantify how gaze patterns may indicate a person's intention. Our investigation was contextualized in a dyadic sandwich-making scenario in which a “worker” prepared a sandwich by adding ingredients requested by a “customer.” In this context, we investigated the extent to which the customers' gaze cues serve as predictors of which ingredients they intend to request. Predictive features were derived to represent characteristics of the customers' gaze patterns. We developed a support vector machine-based (SVM-based) model that achieved 76% accuracy in predicting the customers' intended requests based solely on gaze features. Moreover, the predictor made correct predictions approximately 1.8 s before the spoken request from the customer. We further analyzed several episodes of interactions from our data to develop a deeper understanding of the scenarios where our predictor succeeded and failed in making correct predictions. These analyses revealed additional gaze patterns that may be leveraged to improve intention prediction. This work highlights gaze cues as a significant resource for understanding human intentions and informs the design of real-time recognizers of user intention for intelligent systems, such as assistive robots and ubiquitous devices, that may enable more complex capabilities and improved user experience.

Embodiment in Socially Interactive Robots

“…The gaze cues of an individual, defined by the orientation-and shifts thereof-of the eyes, the head, and the body, convey rich information about the direction of attention and mental and emotional states of the individual Frischen et al (2007). These cues serve a range of social functions, including facilitating turn-taking Duncan (1972); , helping to establish joint attention Emery (2000), and signaling the intent and mental states of others Calder et al (2002); Byom and Mutlu (2013); Huang et al (2015a). The wide range of functions that gaze serves is due largely to their highly contextualized nature.…”

Rebellion and Obedience: The Effects of Intention Prediction in Cooperative Handheld Robots

“…As the prediction model of the picking actions implements the novel aspect of adding the VAPs of related objects, its comparison to existing methods is of particular interest. Figure 7 shows a comparison of our proposed model (where both features F 1 and F 2 are used) to the case where F 1 is the single basis for a prediction such as the model recently explored by Huang et al [11]. It can be seen that both models well exceed the chance of picking randomly.…”

“…An SVM [18] was chosen as a prediction model as this type of supervised machine learning model was used for similar classification problems in the past, e.g. [11]. We divided the sets of VAPs into two categories, one where the associated object was the intended object (labelled as chosen = 1) and another one for the objects that were not chosen for interaction (labelled as chosen = 0).…”

Human intention estimation based on hidden Markov model motion validation for safe flexible robotized warehouses

“…There are also works from the gaming industry perspective, proposing methods for improving the non-playable character's assisting efficiency [18,19]. A driver intention recognition problem was approached in [20] and intention estimation based on gaze data was introduced in [21]. Both of those approaches use learning methods for training the models which has been criticized by [22] emphasizing the drawback of using motion pattern learning techniques for trajectory estimation or intention recognition.…”