A Step Towards Efficient Evaluation of Complex Perception Tasks in Simulation

Abstract

There has been increasing interest in characterising the error behaviour of systems which contain deep learning models before deploying them into any safety-critical scenario. However, characterising such behaviour usually requires large-scale testing of the model that can be extremely computationally expensive for complex real-world tasks. For example, tasks involving compute intensive object detectors as one of their components. In this work, we propose an approach that enables efficient large-scale testing using simplified low-fidelity simulators and without the computational cost of executing expensive deep learning models. Our approach relies on designing an efficient surrogate model corresponding to the compute intensive components of the task under test. We demonstrate the efficacy of our methodology by evaluating the performance of an autonomous driving task in the Carla [8] simulator with reduced computational expense by training efficient surrogate models for PIXOR [40] and CenterPoint [41] LiDAR detectors, whilst demonstrating that the accuracy of the simulation is maintained.Preprint. Under review.

Testing Rare Downstream Safety Violations via Upstream Adaptive Sampling of Perception Error Models

“…First, by considering labels on single obstacles, our model is unable to capture false positive detections. This is a common limitation [2], but could be overcome with a more expressive surrogate. Second, we assume the KITTI auxiliary labels are expressive enough to explain variance in detection performance across the domain.…”

“…Perception Error Models (PEMs) [2] are surrogate models of a perception system which, 1 Craig Innes (craig.innes@ed.ac.uk) and Subramanian Ramamoorthy (s.ramamoorthy@ed.ac.uk) are with the School of Informatics, University of Edinburgh, Edinburgh, United Kingdom. 2 For the purpose of open access, the author(s) has applied a Creative Commons Attribution (CC BY) license to any Accepted Manuscript version arising 3 Work supported by a grant from the UKRI Strategic Priorities Fund to the UKRI Research Node on Trustworthy Autonomous Systems Governance and Regulation (EP/V026607/1, 2020-2024) Fig. 1: The difference in fidelity between real traffic images (top) and simulated images rendered in CARLA (bottom).…”

CoPEM: Cooperative Perception Error Models for Autonomous Driving

“…The authors can also compare their results, as they analyze the same private dataset and have access to both models. More recently, Sadeghi et al [35] adopt PEMs proposed in our previous work [6]. Their implementation relies on a Neural Network trained on synthetic data and a LiDAR-based object detector.…”

“…The objective of PEMs is to inject perception errors into the simulation pipeline. Thus, their validity is achieved if and only if the AV behavior generated embedding a PEM is comparable to the behavior the same AV would generate employing an actual S&P. Sadeghi et al [35] use synthetic data to tune their model. Thus, they can directly compare the resulting behaviors, as the sensing module (synthetic) and the vehicle CyberRT API Fig.…”

Challenges in Virtual Testing of Autonomous Vehicles

“…This field is still under-explored, but a few relevant examples are [38], [39] for the modeling part. Instead, we can find a demonstration of their employment in a simulation pipeline for virtual testing in [36], [40], [41].…”