1D CNN with BLSTM for automated classification of fixations, saccades, and smooth pursuits

Abstract

Deep learning approaches have achieved breakthrough performance in various domains. However, the segmentation of raw eye-movement data into discrete events is still done predominantly either by hand or by algorithms that use handpicked parameters and thresholds. We propose and make publicly available a small 1D-CNN in conjunction with a bidirectional long short-term memory network that classifies gaze samples as fixations, saccades, smooth pursuit, or noise, simultaneously assigning labels in windows of up to 1 s. In addition to unprocessed gaze coordinates, our approach uses different combinations of the speed of gaze, its direction, and acceleration, all computed at different temporal scales, as input features. Its performance was evaluated on a large-scale hand-labeled ground truth data set (GazeCom) and against 12 reference algorithms. Furthermore, we introduced a novel pipeline and metric for event detection in eye-tracking recordings, which enforce stricter criteria on the algorithmically produced events in order to consider them as potentially correct detections. Results show that our deep approach outperforms all others, including the state-of-the-art multi-observer smooth pursuit detector. We additionally test our best model on an independent set of recordings, where our approach stays highly competitive compared to literature methods.

REMoDNaV: Robust Eye-Movement Classification for Dynamic Stimulation

“…The proposed algorithm is rule-based, hence can be applied to data without prior training, apart from the adaptive estimation of velocity thresholds. This aspect distinguishes it from other recent developments based on deep neural networks (Startsev et al, 2018), and machine-learning in general (Zemblys et al, 2018). Some statistical learning algorithms require (labeled) training data, which can be a limitation in the context of a research study.…”

“…The validation analyses presented here are based on three different datasets: a manually annoted dataset (Andersson et al, 2017), and two datasets with prolonged recordings using movie stimuli (Hanke et al, 2016). Beyond our own validation, a recent evaluation of nine different smooth pursuit algorithms by Startsev, Agtzidis and Dorr as part of their recent paper (Startsev et al, 2018) also provides metrics for REMoDNaV. In their analysis, algorithm performance was evaluated against a partially hand-labelled eye movement annotation of the Hollywood2 dataset (Mathe and Sminchisescu, 2012).…”

Characterizing and automatically detecting smooth pursuit in a large-scale ground-truth data set of dynamic natural scenes

“…It was modified in recent works: In Zemblys et al (2019), the events that have the largest intersection are matched (rather than the temporally first intersecting event being treated as a match, as in the original matching scheme of I. T. C. Hooge et al, 2018), and the event-level Cohen's kappa scores are computed accordingly. In Startsev, Agtzidis, and Dorr (2019), a threshold for the ''quality'' of the intersection was recommended, which results in no more than one potential match for each of the ''true'' episodes. In Startsev, Göb, and Dorr, (2019) we additionally proposed a new event-level Cohen's kappa-based statistic, which we developed after analyzing the literature evaluation strategies in the context of eye movement classification baselines.…”

“…To put the performance of our detector in context, we compare it with three other methods that detect SP: the algorithms of Berg et al (2009, implemented in Walther & Koch, 2006 and Larsson et al (2015, reimplemented by our group and available for download on the data repository page), as well as I-VMP (San Agustin, 2010, implemented by Komogortsev, 2014). I-VMP, among others, was optimized in Startsev, Agtzidis, and Dorr (2019) via an exhaustive grid search of its parameters in order to deliver optimal performance on the full GazeCom data set, so its results represent an optimistic scenario. These three models (plus the approach described here) were the best nondeep-learning detectors tested in Startsev, Agtzidis, and Dorr (2019), when ranked by the average per-class sample-and event-level F1 scores.…”

“…Mital et al (2011) then simply combine all ''nonsaccadic eye movements'' into one category. While this is sufficient for some applications, various areas of eye movement research require distinguishing between different ways of looking at the gaze targets, in terms of execution or perception Spering et al, 2011;Silberg et al, 2019).…”