Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset

Abstract

Large-scale public datasets have been shown to benefit research in multiple areas of modern artificial intelligence. For decision-making research that requires human data, highquality datasets serve as important benchmarks to facilitate the development of new methods by providing a common reproducible standard. Many human decision-making tasks require visual attention to obtain high levels of performance. Therefore, measuring eye movements can provide a rich source of information about the strategies that humans use to solve decision-making tasks. Here, we provide a largescale, high-quality dataset of human actions with simultaneously recorded eye movements while humans play Atari video games. The dataset consists of 117 hours of gameplay data from a diverse set of 20 games, with 8 million action demonstrations and 328 million gaze samples. We introduce a novel form of gameplay, in which the human plays in a semiframe-by-frame manner. This leads to near-optimal game decisions and game scores that are comparable or better than known human records. We demonstrate the usefulness of the dataset through two simple applications: predicting human gaze and imitating human demonstrated actions. The quality of the data leads to promising results in both tasks. Moreover, using a learned human gaze model to inform imitation learning leads to an 115% increase in game performance. We interpret these results as highlighting the importance of incorporating human visual attention in models of decision making and demonstrating the value of the current dataset to the research community. We hope that the scale and quality of this dataset can provide more opportunities to researchers in the areas of visual attention, imitation learning, and reinforcement learning.

Self-Supervised Discovering of Interpretable Features for Reinforcement Learning

“…Average return represents the long-term rewards of two policies, while behavior matching characterizes the behavioural similarity of two policies. We note that similar metrics are also used for attention-guided learning in recent work (Zhang et al, 2019).…”

Benchmarking End-to-End Behavioural Cloning on Video Games

“…In addition, if we gather BC data from human players, the recorded actions are subject to delays from human-reflexes: if something surprising happens in an image, average humans react to this with a split-second delay. This action was supposed to be associated with the surprising event, but instead it will be recorded few frames later, associated with possibly a wrong observation and leading to state-action mismatch [14].…”

“…[15], and "how the state-action mismatch from human reflexes affects the results?" [14]. The former sheds light on if we should gather data from only few experts, or should we use data from many different players.…”

“…Apart from [15], related work does not study how data should be chosen for behavioural cloning. In addition, Zhang et al [14] bring up an important point on how human delay can adversarially affect the quality of the dataset but did not include experimental results on this. We aim to answer these three questions.…”

The Paradox of Choice: Using Attention in Hierarchical Reinforcement Learning

“…Learning an attention mechanism that limits these choices could potentially result in much better performance. Humans have a remarkable ability to selectively pay attention to certain parts of the visual input (Judd et al, 2009;Borji et al, 2012), gathering relevant information, and sequentially combining their observations to build representations across different timescales (Hayhoe and Ballard, 2005;Zhang et al, 2019), which plays an important role in guiding further perception and action (Nobre and Stokes, 2019;Badman et al, 2020). In this paper, we explore ideas for endowing reinforcement learning (RL) agents with these type of capabilities.…”