Parameter Coverage for Testing of Autonomous Driving Systems under Uncertainty

Abstract

Autonomous Driving Systems (ADSs) are promising, but must show they are secure and trustworthy before adoption. Simulation-based testing is a widely adopted approach, where the ADS is run in a simulated environment over specific scenarios. Coverage criteria specify what needs to be covered to consider the ADS sufficiently tested. However, existing criteria do not guarantee to exercise the different decisions that the ADS can make, which is essential to assess its correctness. ADSs usually compute their decisions using parameterised rule-based systems and cost functions, such as cost components or decision thresholds. In this paper, we argue that the parameters characterise the decision process, as their values affect the ADS’s final decisions. Therefore, we propose parameter coverage, a criterion requiring to cover the ADS’s parameters. A scenario covers a parameter if changing its value leads to different simulation results, meaning it is relevant for the driving decisions made in the scenario. Since ADS simulators are slightly uncertain, we employ statistical methods to assess multiple simulation runs for execution difference and coverage. Experiments using the Autonomoose ADS show that the criterion discriminates between different scenarios; and that the cost of computing coverage can be managed with suitable heuristics.

Towards Reliable AI: Adequacy Metrics for Ensuring the Quality of System-level Testing of Autonomous Vehicles

“…For this study, we select widely used adequacy metrics for black-box system testing of AI-based systems. The available coverage based metrics are either white-box [34,40,51,64,78], or task specific [9,36,45,79], hence do not fit well for this study. Therefore, all the metrics we selected are diversity-based adequacy measures.…”

“…Diversity is an apparent indicator of quality because it directly indicates how thoroughly the system is tested across various conditions and scenarios. It is a widely used adequacy measure in traditional software [5,9,36,45,53,79] and also deemed important for testing of AI-based systems [13,27,29,50,56]. What makes our work different from these studies is that we propose a novel set of adequacy metrics that assess the quality of a test suite, both in terms of diversity and coverage.…”

“…Tang et al classify scenarios based on map topological structures and evaluate coverage accordingly [79]. Laurent et al [45] use weight coverage to cover different configurations of a path planner. Unlike these studies, our work introduces a coverage metric that provides a quantitative measure of quality.…”

Consistent and Quality-Aware Service Composition in Smart Cities

“…The experiences with these two different directions have established the solid research foundation for the author, that is, investigation of application-level dependability goals with different types of automated techniques. The insights obtained in the experiences have helped the author tackle challenges in different domains such as automated driving systems [22][23][24][25], automated delivery robots [26][27][28], and gamesas-a-service [ 29]. We have been making use of optimization techniques as well as formal verification techniques to deal with various quality aspects though the systems are monolithic, and we focus more on the software engineering aspects such as optimization-based test generation.…”