Human Gaze Assisted Artificial Intelligence: A Review

Abstract

<jats:p>Human gaze reveals a wealth of information about internal cognitive state. Thus, gaze-related research has significantly increased in computer vision, natural language processing, decision learning, and robotics in recent years. We provide a high-level overview of the research efforts in these fields, including collecting human gaze data sets, modeling gaze behaviors, and utilizing gaze information in various applications, with the goal of enhancing communication between these research areas. We discuss future challenges and potential applications that work towards a common goal of human-centered artificial intelligence.</jats:p>

Gazing at Social Interactions Between Foraging and Decision Theory

“…The computational efficiency of the method shows promise for application in robotics, markedly in social robotics, where active vision plays an important role and where social robot's sensitivity to environmental information and the ability to localize the people around itself is crucial (Admoni and Scassellati, 2017 ; Wiese et al, 2017 ; Zhang et al, 2020 ). Social robots need to gather information about their human fellows to facilitate mutual understanding and coordination (Zhang et al, 2020 ). Designing robot gaze itself is challenging and difficult to standardize due to the variations in physical robots and human participants, while burdened with architectural constraints.…”

“…Designing robot gaze itself is challenging and difficult to standardize due to the variations in physical robots and human participants, while burdened with architectural constraints. Early research efforts (Breazeal et al, 2001 ) relied on simple saliency-based schemes (Itti et al, 1998 ) inherited from computer vision (Shic and Scassellati, 2007 ; Ferreira and Dias, 2014 ); in the last decade these have been reshaped in the form of deep neural nets, such as convolutional networks (Zhang et al, 2020 ). Yet, the aptness of accounting for task, value and context in the visuo-motor loop is crucial.…”

“…In our setting, no specific external task or goal is given (free-viewing condition). Then, if the ultimate objective of an active perceiver is total reward maximization (Zhang et al, 2020 ), reward can be related to the “internal” value (Berridge and Robinson, 2003 ). The latter has different psychological facets including affect (implicit “liking” and conscious pleasure) and motivation (implicit incentive salience, “wanting”).…”

“…Yet, even when limiting to the unimodal case of visual stimuli, gaze dynamics has been by and large overlooked in computer vision in spite of the pioneering work of Aloimonos et al ( 1988 ), Ballard ( 1991 ), and Bajcsy and Campos ( 1992 ). The current state of affairs is that effort is mostly spent to model salience (Borji and Itti, 2013 ; Borji, 2021 ) as a tool for predicting where/what to look at (for a critical discussion, see Tatler et al, 2011 ; Le Meur and Liu, 2015 ; Foulsham, 2019 ; Boccignone et al, 2020 ; Zhang et al, 2020 ).…”

Leveraging Human Guidance for Deep Reinforcement Learning Tasks

“…This allows humans to move their foveae to the right place at the right time, in order to attend to important visual features (Diaz et al 2013). Therefore, human expert's gaze serves as a good standard in many vision-related tasks for evaluating machine attention, or as learning target for training machine attention (Qiuxia et al 2020;Zhang et al 2020a). This approach is widely used in the computer vision research, see Nguyen et al (Nguyen, Zhao, and Yan 2018) for a review.…”