Is human classification by experienced untrained observers a gold standard in fixation detection?

Hooge, Ignace T. C.; Niehorster, Diederick C; Nyström, Maria; Andersson, Richard; Hessels, Roy S.

doi:10.3758/s13428-017-0955-x

Cited by 60 publications

(95 citation statements)

References 54 publications

(86 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…O r is the overlap ratio between matched events. l 2 distance between matched event start and end times and their standard deviation l 2 − σ in ms. l 2 and l 2 − σ are similar to RTO and RTD metrics proposed by Hooge et al 44…”

Section: Resultssupporting

confidence: 72%

“…Evaluating the performance of automated classification systems or human labellers is not straightforward. 46 Sample-level majority vote × × × N/A Event F1 44 Earliest overlapping event × × × low Event kappa 21 Largest overlapping event × low Event error rate 21 N/A × × × N/A ELC Window-based matching × high Table 4. Comparison of event level error metrics on a global basis, thus oblivious to the inherent structure of the data.…”

Section: Error Metricsmentioning

confidence: 99%

See 1 more Smart Citation

Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities

Kothari

Yang²,

Kanan³

et al. 2020

Sci Rep

View full text Add to dashboard Cite

The interaction between the vestibular and ocular system has primarily been studied in controlled environments. Consequently, off-the shelf tools for categorization of gaze events (e.g. fixations, pursuits, saccade) fail when head movements are allowed. Our approach was to collect a novel, naturalistic, and multimodal dataset of eye+head movements when subjects performed everyday tasks while wearing a mobile eye tracker equipped with an inertial measurement unit and a 3D stereo camera. This Gaze-in-the-Wild dataset (GW) includes eye+head rotational velocities (deg/s), infrared eye images and scene imagery (RGB+D). A portion was labelled by coders into gaze motion events with a mutual agreement of 0.72 sample based Cohen's κ. This labelled data was used to train and evaluate two machine learning algorithms, Random Forest and a Recurrent Neural Network model, for gaze event classification. Assessment involved the application of established and novel event based performance metrics. Classifiers achieve ∼90% human performance in detecting fixations and saccades but fall short (60%) on detecting pursuit movements. Moreover, pursuit classification is far worse in the absence of head movement information. A subsequent analysis of feature significance in our best-performing model revealed a reliance upon absolute eye and head velocity, indicating that classification does not require spatial alignment of the head and eye tracking coordinate systems. The GW dataset, trained classifiers and evaluation metrics will be made publicly available with the intention of facilitating growth in the emerging area of head-free gaze event classification. arXiv:1905.13146v1 [cs.CV] 9 May 2019 (see Section 3). We then use this labelled data for supervised training and assessment of automated event detectors.This work builds upon a variety of techniques previously used to track head orientation during natural behavior. Published studies have demonstrated the use of rotational potentiometers and accelerometers, 8 magnetic coils, 12 or motion capture 13 for the sensing of head orientation. 6 Perhaps the highest precision eye+head tracker which allowed body movement leveraged a 5.8 m 3 custom-made armature capable of generating a pulsing magnetic field. The subject was outfitted with a head-worn receiver capable of measuring head position and orientation within its operational region. 14 Several systems have adopted video based head motion compensation 10, 15 and demonstrated promising results, but are too computationally expensive for real-time use, and are prone to irrecoverable track loss especially during periods of rapid head movement, occlusion of tracking features or degration of image quality due to motion blur. Recent approaches have involved the use of head-mounted IMUs. For example, Larsson et al. used a head-mounted IMU in a study where subjects were asked to perform visual tracking tasks when watching pre-rendered stimuli projected onto a 2D screen. 16 They established that compensating for head movements results in a reduced sta...

show abstract

Section: Resultssupporting

confidence: 72%

Section: Error Metricsmentioning

confidence: 99%

Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities

Kothari

Yang²,

Kanan³

et al. 2020

Sci Rep

View full text Add to dashboard Cite

show abstract

“…In Hooge, Niehorster, Nyström, Andersson, and Hessels (2017), event-level fixation detection was assessed by an arguably fairer approach with a set of metrics that includes F1 scores for fixation episodes. We computed these for all three main event types in our data (fixations, saccades, and smooth pursuits): For each event in the ground truth, we look for the earliest algorithmically detected event of the same class that intersects with it.…”

Section: Metricsmentioning

confidence: 99%

“…Automatically detecting different eye movements has been attempted for multiple decades by now, but evaluating the approaches for this task is challenging, not least because of the diversity of the data and the amount of manual labeling required for a meaningful evaluation. To compound this problem, even manual annotations suffer from individual biases and implicitly used thresholds and rules, especially if experts from different sub-areas are involved (Hooge, Niehorster, Mikhail Startsev mikhail.startsev@tum Nyström, Andersson, & Hessels, 2017). For smooth pursuit (SP), even detecting episodes 1 by hand is not entirely trivial (i.e., requires additional information) when the information about their targets is missing.…”

Section: Introductionmentioning

confidence: 99%

1D CNN with BLSTM for automated classification of fixations, saccades, and smooth pursuits

2018

View full text Add to dashboard Cite

Deep learning approaches have achieved breakthrough performance in various domains. However, the segmentation of raw eye-movement data into discrete events is still done predominantly either by hand or by algorithms that use handpicked parameters and thresholds. We propose and make publicly available a small 1D-CNN in conjunction with a bidirectional long short-term memory network that classifies gaze samples as fixations, saccades, smooth pursuit, or noise, simultaneously assigning labels in windows of up to 1 s. In addition to unprocessed gaze coordinates, our approach uses different combinations of the speed of gaze, its direction, and acceleration, all computed at different temporal scales, as input features. Its performance was evaluated on a large-scale hand-labeled ground truth data set (GazeCom) and against 12 reference algorithms. Furthermore, we introduced a novel pipeline and metric for event detection in eye-tracking recordings, which enforce stricter criteria on the algorithmically produced events in order to consider them as potentially correct detections. Results show that our deep approach outperforms all others, including the state-of-the-art multi-observer smooth pursuit detector. We additionally test our best model on an independent set of recordings, where our approach stays highly competitive compared to literature methods.

show abstract

“…Previous research has shown that eye-tracking researchers may set different thresholds as to what constitutes a fixation-breaking eye movement [42]. …”

Section: Methodsmentioning

confidence: 99%

Is the eye-movement field confused about fixations and saccades? A survey among 124 researchers

et al. 2018

Self Cite

View full text Add to dashboard Cite

Eye movements have been extensively studied in a wide range of research fields. While new methods such as mobile eye tracking and eye tracking in virtual/augmented realities are emerging quickly, the eye-movement terminology has scarcely been revised. We assert that this may cause confusion about two of the main concepts: fixations and saccades. In this study, we assessed the definitions of fixations and saccades held in the eye-movement field, by surveying 124 eye-movement researchers. These eye-movement researchers held a variety of definitions of fixations and saccades, of which the breadth seems even wider than what is reported in the literature. Moreover, these definitions did not seem to be related to researcher background or experience. We urge researchers to make their definitions more explicit by specifying all the relevant components of the eye movement under investigation: (i) the oculomotor component: e.g. whether the eye moves slow or fast; (ii) the functional component: what purposes does the eye movement (or lack thereof) serve; (iii) the coordinate system used: relative to what does the eye move; (iv) the computational definition: how is the event represented in the eye-tracker signal. This should enable eye-movement researchers from different fields to have a discussion without misunderstandings.

show abstract

Is human classification by experienced untrained observers a gold standard in fixation detection?

Cited by 60 publications

References 54 publications

Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities

Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities

1D CNN with BLSTM for automated classification of fixations, saccades, and smooth pursuits

Is the eye-movement field confused about fixations and saccades? A survey among 124 researchers

Contact Info

Product

Resources

About