An Examination of Recording Accuracy and Precision From Eye Tracking Data From Toddlerhood to Adulthood

Abstract

The quantitative assessment of eye tracking data quality is critical for ensuring accuracy and precision of gaze position measurements. However, researchers often report the eye tracker’s optimal manufacturer’s specifications rather than empirical data about the accuracy and precision of the eye tracking data being presented. Indeed, a recent report indicates that less than half of eye tracking researchers surveyed take the eye tracker’s accuracy into account when determining areas of interest for analysis, an oversight that could impact the validity of reported results and conclusions. Accordingly, we designed a calibration verification protocol to augment independent quality assessment of eye tracking data and examined whether accuracy and precision varied between three age groups of participants. We also examined the degree to which our externally quantified quality assurance metrics aligned with those reported by the manufacturer. We collected data in standard laboratory conditions to demonstrate our method, to illustrate how data quality can vary with participant age, and to give a simple example of the degree to which data quality can differ from manufacturer reported values. In the sample data we collected, accuracy for adults was within the range advertised by the manufacturer, but for school-aged children, accuracy and precision measures were outside this range. Data from toddlers were less accurate and less precise than data from adults. Based on an a priori inclusion criterion, we determined that we could exclude approximately 20% of toddler participants for poor calibration quality quantified using our calibration assessment protocol. We recommend implementing and reporting quality assessment protocols for any eye tracking tasks with participants of any age or developmental ability. We conclude with general observations about our data, recommendations for what factors to consider when establishing data inclusion criteria, and suggestions for stimulus design that can help accommodate variability in calibration. The methods outlined here may be particularly useful for developmental psychologists who use eye tracking as a tool, but who are not experts in eye tracking per se. The calibration verification stimuli and data processing scripts that we developed, along with step-by-step instructions, are freely available for other researchers.

Comparing Automatic Eye Tracking and Manual Gaze Coding Methods in Young Children with Autism Spectrum Disorder

“…However, the fact that automatic eye tracking is based on light reflections and automated algorithms instead of human judgment does not mean it is always accurate. A growing number of studies have begun to identify concerns regarding the accuracy of automatic eye trackers, especially in populations that may demonstrate considerable head and body movement [Dalrymple et al, ; Hessels, Andersson, et al, ; Hessels, Cornelissen, et al, ; Niehorster et al, ; Schlegelmilch & Wertz, ].…”

“…Dalrymple et al [] examined the accuracy of an automated Tobii eye tracker and found that data from toddlers with typical development had poorer accuracy and precision than data from school‐aged children and adults. In fact, the mean accuracy for toddlers fell outside the accuracy range described in the eye‐tracker manual.…”

“…High‐quality calibration increases accuracy, but it can be difficult to achieve [Aslin & McMurray, ; Nyström et al, ; Schlegelmilch & Wertz, ; Tenenbaum, Amso, Abar, & Sheinkopf, ]. Some children may be unable to complete calibration [Dalrymple et al, ], and time spent on calibration (and re‐calibration) decreases the likelihood that children will remain engaged in the remainder of the task [Aslin & McMurray, ]. It can also be difficult to tell whether poor calibration occurs because of the measurement error of the system, or because a child did not actually fixate the intended target.…”

“…Because of its automated approach, eye tracking offers several advantages over manual gaze coding. It is objective, efficient, and has relatively high temporal and spatial resolution [Dalrymple, Manner, Harmelink, Teska, & Elison, ; Hessels, Andersson, Hooge, Nyström, & Kemner, ]. Manual gaze coding, on the other hand, is subjective, requires extensive reliability training, is labor‐intensive (e.g., about 1 hr for a 5‐min video), and has more limited spatial and temporal resolution [Aslin & McMurray, ; Wass et al, ].…”

“…In manual gaze coding, for example, a human coder could judge a child to be looking at an image AOI, when in fact the child is looking slightly outside the boundaries of the AOI (e.g., off-screen or at a non-AOI part of the screen). The same type of error could occur in automatic eye tracking when internal processing algorithms estimate gaze location inaccurately [Dalrymple et al, 2018;Niehorster, Cornelissen, Holmqvist, Hooge, & Hessels, 2018]. Because both systems are capable of reporting gaze location inaccurately, we do not consider inaccuracy to be a disadvantage unique to either system.…”