Uncertainty-sensitive Activity Recognition: A Reliability Benchmark and the CARING Models

Abstract

Beyond assigning the correct class, an activity recognition model should also be able to determine, how certain it is in its predictions. We present the first study of how well the confidence values of modern action recognition architectures indeed reflect the probability of the correct outcome and propose a learning-based approach for improving it. First, we extend two popular action recognition datasets with a reliability benchmark in form of the expected calibration error and reliability diagrams. Since our evaluation highlights that confidence values of standard action recognition architectures do not represent the uncertainty well, we introduce a new approach which learns to transform the model output into realistic confidence estimates through an additional calibration network. The main idea of our Calibrated Action Recognition with Input Guidance (CARING) model is to learn an optimal scaling parameter depending on the video representation. We compare our model with the native action recognition networks and the temperature scaling approach -a wide spread calibration method utilized in image classification. While temperature scaling alone drastically improves the reliability of the confidence values, our CARING method consistently leads to the best uncertainty estimates in all benchmark settings.

Should I take a walk? Estimating Energy Expenditure from Video Data

“…Fueled by multiple publicly released large-scale activity recognition datasets collected from Youtube/Movies [25,29,53] or in home environments [50], the research of deep learning based activity recognition became a very active research field also explored in more targeted applications, e.g., in cooking [11,46], sports [42], robotics [23,49], and automated driving [35]-related tasks. More specialized activity recognition research also addressed topics such as uncertainty of video classification models [47,54]. However, all the approaches focus on categorization into previously defined activity classes, while examining their feasibility for capturing complex physiological processes of the body, such as our calorie expenditure task, has been largely overlooked.…”

Is my Driver Observation Model Overconfident? Input-guided Calibration Networks for Reliable and Interpretable Confidence Estimates

“…This paper is an extension of our conference publication [35], which has been expanded with specific focus on driver behavior understanding, a detailed description of the proposed CARING method, and an extended set of experiments and analyses.…”