Handling annotation uncertainty in human activity recognition

Kwon, Hyeokhyen; Abowd, Gregory D.; Plötz, Thomas

doi:10.1145/3341163.3347744

Cited by 35 publications

(7 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, due to the absence of an affective response, the normal windows are same as the distracted windows when analyzed using the physiological and the visual sensors. To mitigate this problem, one might use methods that explicitly incorporate label jitter into the model training process [78]. The label jitter may be why the performance on the physiological signals is worse than that on the visual signals for the detection of the driving distractions.…”

Section: Discussionmentioning

confidence: 99%

Machine Learning and End-to-End Deep Learning for Monitoring Driver Distractions From Physiological and Visual Signals

et al. 2020

View full text Add to dashboard Cite

It is only a matter of time until autonomous vehicles become ubiquitous; however, human driving supervision will remain a necessity for decades. To assess the driver's ability to take control over the vehicle in critical scenarios, driver distractions can be monitored using wearable sensors or sensors that are embedded in the vehicle, such as video cameras. The types of driving distractions that can be sensed with various sensors is an open research question that this study attempts to answer. This study compared data from physiological sensors (palm electrodermal activity (pEDA), heart rate and breathing rate) and visual sensors (eye tracking, pupil diameter, nasal EDA (nEDA), emotional activation and facial action units (AUs)) for the detection of four types of distractions. The dataset was collected in a previous driving simulation study. The statistical tests showed that the most informative feature/modality for detecting driver distraction depends on the type of distraction, with emotional activation and AUs being the most promising. The experimental comparison of seven classical machine learning (ML) and seven end-to-end deep learning (DL) methods, which were evaluated on a separate test set of 10 subjects, showed that when classifying windows into distracted or not distracted, the highest F1-score of 79% was realized by the extreme gradient boosting (XGB) classifier using 60-second windows of AUs as input. When classifying complete driving sessions, XGB's F1-score was 94%. The best-performing DL model was a spectro-temporal ResNet, which realized an F1-score of 75% when classifying segments and an F1-score of 87% when classifying complete driving sessions. Finally, this study identified and discussed problems, such as label jitter, scenario overfitting and unsatisfactory generalization performance, that may adversely affect related ML approaches.

show abstract

Section: Discussionmentioning

confidence: 99%

Machine Learning and End-to-End Deep Learning for Monitoring Driver Distractions From Physiological and Visual Signals

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Our current practice of using majority voting is also unable to deal with the situation when multiple activities exist on the same frame. For this problem, we plan to follow the recent progress in ambiguous activity annotation [33] to drive our future research. 6.1.2 Limited Interaction between HAR and PBD Modules.…”

Section: The Challenge and Current Limitationsmentioning

confidence: 99%

“…In particular, discriminating the margin between activity-of-interest and transition is often difficult and may lead to the misclassification of transition toward AoIs, as shown in Section 5.3. We plan to follow recent progress on ambiguous activity annotation [31] to deal with such uncertainty in the next.…”

Section: The Challenge and Current Limitationsmentioning

confidence: 99%

Leveraging Activity Recognition to Enable Protective Behavior Detection in Continuous Data

Wang

Gao

Mathur

et al. 2021

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

Protective behavior exhibited by people with chronic pain (CP) during physical activities is very informative to understanding their physical and emotional states. Existing automatic protective behavior detection (PBD) methods rely on pre-segmentation of activities predefined by users. However, in real life, people perform activities casually. Therefore, where those activities present difficulties for people with CP, technology-enabled support should be delivered continuously and automatically adapted to activity type and occurrence of protective behavior. Hence, to facilitate ubiquitous CP management, it becomes critical to enable accurate PBD over continuous data. In this paper, we propose to integrate human activity recognition (HAR) with PBD via a novel hierarchical HAR-PBD architecture comprising graph-convolution and long short-term memory (GC-LSTM) networks, and alleviate class imbalances using a class-balanced focal categorical cross-entropy (CFCC) loss. Through in-depth evaluation of the approach using a CP patients' dataset, we show that the leveraging of HAR, GC-LSTM networks, and CFCC loss leads to clear increase in PBD performance against the baseline (macro F1 score of 0.81 vs. 0.66 and precision-recall area-under-the-curve (PR-AUC) of 0.60 vs. 0.44). We conclude by discussing possible use cases of the hierarchical architecture in CP management and beyond. We also discuss current limitations and ways forward.

show abstract

“…Such outlier samples impose, if not treated properly, challenges for HAR models to learn effective motion features for target activities. Previous work showed that handling outlier samples in HAR datasets is important and, if successful, can significantly improve the performance of the resulting HAR model [91][92][93].…”

Section: Uncertainty Modeling For Noisy Samplesmentioning

confidence: 99%

Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables

Kwon

Abowd

Plötz

2021

Sensors

Self Cite

View full text Add to dashboard Cite

Supervised training of human activity recognition (HAR) systems based on body-worn inertial measurement units (IMUs) is often constrained by the typically rather small amounts of labeled sample data. Systems like IMUTube have been introduced that employ cross-modality transfer approaches to convert videos of activities of interest into virtual IMU data. We demonstrate for the first time how such large-scale virtual IMU datasets can be used to train HAR systems that are substantially more complex than the state-of-the-art. Complexity is thereby represented by the number of model parameters that can be trained robustly. Our models contain components that are dedicated to capture the essentials of IMU data as they are of relevance for activity recognition, which increased the number of trainable parameters by a factor of 1100 compared to state-of-the-art model architectures. We evaluate the new model architecture on the challenging task of analyzing free-weight gym exercises, specifically on classifying 13 dumbbell execises. We have collected around 41 h of virtual IMU data using IMUTube from exercise videos available from YouTube. The proposed model is trained with the large amount of virtual IMU data and calibrated with a mere 36 min of real IMU data. The trained model was evaluated on a real IMU dataset and we demonstrate the substantial performance improvements of 20% absolute F1 score compared to the state-of-the-art convolutional models in HAR.

show abstract

Handling annotation uncertainty in human activity recognition

Cited by 35 publications

References 27 publications

Machine Learning and End-to-End Deep Learning for Monitoring Driver Distractions From Physiological and Visual Signals

Machine Learning and End-to-End Deep Learning for Monitoring Driver Distractions From Physiological and Visual Signals

Leveraging Activity Recognition to Enable Protective Behavior Detection in Continuous Data

Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables

Contact Info

Product

Resources

About