From Evaluation to Verification: Towards Task-oriented Relevance Metrics for Pedestrian Detection in Safety-critical Domains

Abstract

Whenever a visual perception system is employed in safety-critical applications such as automated driving, a thorough, task-oriented experimental evaluation is necessary to guarantee safe system behavior. While most standard evaluation methods in computer vision provide a good comparability on benchmarks, they tend to fall short on assessing the system performance that is actually relevant for the given task. In our work, we consider pedestrian detection as a highly relevant perception task, and we argue that standard measures such as Intersection over Union (IoU) give insufficient results, mainly because they are insensitive to important physical cues including distance, speed, and direction of motion. Therefore, we investigate so-called relevance metrics, where specific domain knowledge is exploited to obtain a task-oriented performance measure focusing on distance in this initial work. Our experimental setup is based on the CARLA simulator and allows a controlled evaluation of the impact of that domain knowledge. Our first results indicate a linear decrease of the IoU related to the pedestrians' distance, leading to the proposal of a first relevance metric that is also conditioned on the distance.

Semantic Concept Testing in Autonomous Driving by Extraction of Object-Level Annotations from CARLA

“…we cannot generate pedestrian-specific metadata. As also discussed in Lyssenko et al [9] this lacking separability and identifiability of individual pedestrians can be challenging for some more advanced analysis (depthbased in their case) and requires a workaround.…”

“…There is also no issue of domain gap as training, and test data are from the same domain. Parallel to our work, Lyssenko et al [9] present a CARLA-based extension to perform validation of DNNs by considering the pedestrian distance from the ego vehicle as a safety relevance metric. However, they mention that their bounding box retrieval method for getting instances has some issues.…”

Low-cost system for analysis pedestrian flow from an aerial view using Near-Infrared, Microwave, and Temperature sensors

“…The determination in the use of Device-Free sensors, allows the independence of the system in the collection of data for the identification of the passer-by. The need to establish and depend on equipping the pedestrian with a device segments the type of public to be accounted for [9] , [10] .…”

Interaction-Dynamics-Aware Perception Zones for Obstacle Detection Safety Evaluation

“…The misalignment between typical perception metrics and application makes it challenging to rely on existing metrics to verify and validate perception algorithms deployed in the real-world. In fact, using a case study on a pedestrian detection algorithm on an AV, [11] showed there was a linear degradation in IoU performance the further away a pedestrian was, therefore indicating that IoU alone is not sufficient in validating safety. As such, they propose two different evaluation metrics that require user-specified inputs.…”