Does agreement mean accuracy? Evaluating glance annotation in naturalistic driving data

Abstract

Naturalistic driving studies often make use of cameras to monitor driver behavior. To analyze the resulting video images, human annotation is often adopted. These annotations then serve as the 'gold standard' to train and evaluate automated computer vision algorithms, even though it is uncertain how accurate human annotation is. In this study, we provide a first evaluation of glance direction annotation by comparing instructed, actual glance direction of truck drivers with annotated direction. Findings indicate that while for some locations high annotation accuracy is achieved, for most locations accuracy is well below 50%. Higher accuracy can be obtained by clustering these locations, but this also leads to reduced detail of the annotation, suggesting that decisions to use clustering should take into account the purpose of the annotation. The data also show that high agreement between annotators does not guarantee high accuracy. We argue that the accuracy of annotation needs to be verified experimentally more often.

Attention for Vision-Based Assistive and Automated Driving: A Review of Algorithms and Datasets

“…For example, determining specific objects or areas that the driver is observing is not trivial. A recent study by Jansen et al [256] raised concerns regarding the manual annotation of gaze from driver-facing videos. Based on their analysis, the customary practice of measuring several independent annotators' agreement may not produce good quality labels as some areas of interest are easily confused (e.g.…”

On Driver Behavior Recognition for Increased Safety: A Roadmap

“…On the other hand, physiological signals generally start changing in the earlier stages of dangerous situations and, therefore, physiological signals can be employed for anomalous behavior detection, with a negligible false positive rate [154]. Analyzing the collected raw physiological data should take into account the noise and artifacts related to all movements made by drivers during driving, but the reliability and accuracy of driver behavior estimation is higher compared to other methods [155]. One of the main inhibitors for the large adoption of methodologies based on physiological signals is related to the typically obtrusive nature of their current implementations.…”

A Client-Cloud-Chain Data Annotation System of Internet of Things for Semi-Supervised Missing Data

“…Data annotation is an important task for data preprocessing and knowledge acquisition in the data center to ensure efficiency in data-enabled businesses [1][2][3][4]. In recent years, there have been many studies on automatic annotation, but its accuracy and applicability cannot meet engineering requirements [5][6][7]. High-precision data labels can help people use data more conveniently and efficiently, such as training models, fast and accurate data positioning [8], etc.…”