Confidence‐driven weighted retraining for predicting safety‐critical failures in autonomous driving systems

Abstract

Safe handling of hazardous driving situations is a task of high practical relevance for building reliable and trustworthy cyber‐physical systems such as autonomous driving systems. This task necessitates an accurate prediction system of the vehicle's confidence to prevent potentially harmful system failures on the occurrence of unpredictable conditions that make it less safe to drive. In this paper, we discuss the challenges of adapting a misbehavior predictor with knowledge mined during the execution of the main system. Then, we present a framework for the continual learning of misbehavior predictors, which records in‐field behavioral data to determine what data are appropriate for adaptation. Our framework guides adaptive retraining using a novel combination of in‐field confidence metric selection and reconstruction error‐based weighing. We evaluate our framework to improve a misbehavior predictor from the literature on the Udacity simulator for self‐driving cars. Our results show that our framework can reduce the false positive rate by a large margin and can adapt to nominal behavior drifts while maintaining the original capability to predict failures up to several seconds in advance.

Efficient and Effective Feature Space Exploration for Testing Deep Learning Systems

“…However, since these metrics characterise only the inputs' structural features, we considered additional metrics to characterise the quality of self-driving, i.e., metrics to quantify the behavioural features. In particular, we selected the following two metrics from the study by Jahangirova et al [35]: standard deviation of steering angle (StdSA), which measures the activity of the driving agent on the steering wheel and can be used to quantify passenger comfort; and the car's mean lateral position (MLP), which measures how close the driving agent drives from to the lane margins and can be used to measure safety [22,59,63].…”

Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems

“…We measure predictive uncertainty with the variance of dropout-based DNNs' predictions, estimated using the Monte Carlo (MC) method, or MC-Dropout [41]. MC-Dropout approximates epistemic uncertainty of DNNs that perform a regression, such as our SDC models [38], [42].…”

“…While we also use (universal) adversarial attacks, differently, in our work, we focus on system-level testing and on simulated vs real SDCs, finding simulated SDCs generally more susceptible to adversarial attacks than their real-world counterparts when tested at the system level. Concerning system-level testing techniques for AVs, researchers proposed techniques to generate scenarios that cause AVs to misbehave [11], [15], [42], [50], [57]. These works only consider simulated testing, whereas we compared virtual vs physical environments.…”

About the special issue on: “Distributed Complex Systems: Governance, Engineering, and Maintenance”

“…The paper “ Confidence‐driven Weighted Retraining for Predicting Safety‐Critical Failures in Autonomous Driving Systems ” 1 addresses the challenges of predicting potential harmful failures when unexpected system conditions occur. The work focuses on autonomous driving systems and proposes a framework that guides the adaptive retraining of misbehavior predictors.…”