Observation Based Creation of Minimal Test Suites for Autonomous Vehicles

Wolschke, Christian; Kuhn, Thomas S.; Rombach, Dieter; Liggesmeyer, Peter

doi:10.1109/issrew.2017.46

Cited by 20 publications

(22 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…7 Testing Levels i.e., in principle it may be applicable to any MLS (Aniculaesei et al 2018;Byun et al 2019;Cheng et al 2018a, b;Du et al 2019;Eniser et al 2019;Guo et al 2018;Henriksson et al 2019;Kim et al 2019;Li et al 2018;Ma et al 2018bMa et al , c, d, 2019Murphy et al 2007aMurphy et al , b, 2008Murphy et al , b, 2009Nakajima and Bui 2016, 2019Odena et al 2019;Pei et al 2017;Saha and Kanewala 2019;Sekhon and Fleming 2019;Shen et al 2018;Shi et al 2019;Sun et al 2018a, b;Tian et al 2018;Udeshi and Chattopadhyay 2019;Uesato et al 2019;Xie et al 2018Xie et al , 2019Xie et al , 2011Zhang et al 2018aZhang et al , 2019Zhao and Gao 2018). Around 30% proposed approaches are designed for autonomous systems (Abeysirigoonawardena et al 2019;Beglerovic et al 2017;Bühler and Wegener 2004;Klueck et al 2018;Li et al 2016;Mullins et al 2018;de Oliveira Neves et al 2016;Patel et al 2018;Strickland et al 2018;Wolschke et al 2017;Fremont et al 2019), am...…”

Section: Domains (Rq 13)mentioning

confidence: 99%

“…Nearly 60% (33 out of 55) of the papers present various methods to generate test inputs for the MLS Fig. 10 Test Artefacts under test (Abdessalem et al 2016(Abdessalem et al , 2018aAbeysirigoonawardena et al 2019;Beglerovic et al 2017;Bühler and Wegener 2004;Du et al 2019;Eniser et al 2019;Fremont et al 2019;Guo et al 2018;Klueck et al 2018;Li et al 2016;Ma et al 2018cMa et al , 2019Majumdar et al 2019;Murphy et al 2007a, b;de Oliveira Neves et al 2016;Odena et al 2019;Pei et al 2017;Sekhon and Fleming 2019;Sun et al 2018a, b;Tian et al 2018;Tuncali et al 2018Tuncali et al , 2019Udeshi et al 2018Udeshi et al , 2019Wolschke et al 2017Wolschke et al , 2018Xie et al 2019;Zhang et al 2018b;Zheng et al 2018b). However, what a test input represents differs across the proposed approaches and highly depends on the domain of the tested system.…”

Section: Test Artefacts (Rq 21)mentioning

confidence: 99%

“…The proposed testing approaches require different levels of access to the MLS under test. In 29 cases out of 70 (41%), it is enough to have a black-box access to the system (Abdessalem et al 2016(Abdessalem et al , 2018aAbeysirigoonawardena et al 2019;Aniculaesei et al 2018;Beglerovic et al 2017;Bolte et al 2019;Bühler and Wegener 2004;Xie et al 2018;Fremont et al 2019;Klueck et al 2018;Li et al 2016;Majumdar et al 2019;Mullins et al 2018;Murphy et al , 2009Nakajima 2018;de Oliveira Neves et al 2016;Patel et al 2018;Qin et al 2018;Rubaiyat et al 2018;Tuncali et al 2018Tuncali et al , 2019Udeshi and Chattopadhyay 2019;Uesato et al 2019;Wolschke et al 2018;Wolschke et al 2017;Zhao and Gao 2018;Zheng et al 2019;Zhang et al 2016). In 11 cases (16%), along with the inputs and outputs of MLS, training data should also be available: black-box access is not sufficient and data-box access to the system should be provided (Cheng et al 2018b;Ding et al 2017;Henriksson et al 2019;Saha and Kanewala 2019;Udeshi et al 2018;Xie et al 2011;Dwarakanath et al 2018;Groce et al 2014;Zhang et al 2018b;Murphy et al 2007b;…”

Section: Access To the System (Rq 25)mentioning

confidence: 99%

“…In our pool of papers, 17 works (24%) model the context in which the MLS operates. All of them are in the autonomous driving domain (Abdessalem et al 2016, b, Abeysirigoonawardena et al 2019Beglerovic et al 2017;Bolte et al 2019;Bühler and Wegener 2004;Cheng et al 2018b;Fremont et al 2019;Klueck et al 2018;Li et al 2016;Mullins et al 2018;de Oliveira Neves et al 2016;Tuncali et al 2018Tuncali et al , 2019Wolschke et al 2017;Zhao and Gao 2018). The context models presented in these papers vary in terms of simplicity and number of considered actors.…”

Section: Context Model (Rq 26)mentioning

confidence: 99%

See 3 more Smart Citations

Testing machine learning based systems: a systematic mapping

et al. 2020

View full text Add to dashboard Cite

Context: A Machine Learning based System (MLS) is a software system including one or more components that learn how to perform a task from a given data set. The increasing adoption of MLSs in safety critical domains such as autonomous driving, healthcare, and finance has fostered much attention towards the quality assurance of such systems. Despite the advances in software testing, MLSs bring novel and unprecedented challenges, since their behaviour is defined jointly by the code that implements them and the data used for training them. Objective: To identify the existing solutions for functional testing of MLSs, and classify them from three different perspectives: (1) the context of the problem they address, (2) their features, and (3) their empirical evaluation. To report demographic information about the ongoing research. To identify open challenges for future research. Method: We conducted a systematic mapping study about testing techniques for MLSs driven by 33 research questions. We followed existing guidelines when defining our research protocol so as to increase the repeatability and reliability of our results. Results: We identified 70 relevant primary studies, mostly published in the last years. We identified 11 problems addressed in the literature. We investigated multiple aspects of the testing approaches, such as the used/proposed adequacy criteria, the algorithms for test input generation, and the test oracles. Conclusions: The most active research areas in MLS testing address automated scenario/input generation and test oracle creation. MLS testing is a rapidly growing and developing research area, with many open challenges, such as the generation of realistic inputs and the definition of reliable evaluation metrics and benchmarks.

show abstract

Section: Domains (Rq 13)mentioning

confidence: 99%

Section: Test Artefacts (Rq 21)mentioning

confidence: 99%

Section: Access To the System (Rq 25)mentioning

confidence: 99%

Section: Context Model (Rq 26)mentioning

confidence: 99%

See 2 more Smart Citations

Testing machine learning based systems: a systematic mapping

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The results of their experiment indicate that in specific crash scenarios, utilitarian ethics may reduce the total number of fatalities that result from automated vehicle crashes. [66] proposed an approach to describe test-cases for validating autonomous vehicles using recordings of traffic situations for creating a minimal test-suit that could help in the certification process. Considering the example presented, they show how minimalism is achieved by manually comparing the test-cases.…”

Section: Reported Findings (Rq5)mentioning

confidence: 99%

A Systematic Literature Review About the Impact of Artificial Intelligence on Autonomous Vehicle Safety

Nascimento

Vismari

Molina

et al. 2020

IEEE Trans. Intell. Transport. Syst.

View full text Add to dashboard Cite

Autonomous Vehicles (AV) are expected to bring considerable benefits to society, such as traffic optimization and accidents reduction. They rely heavily on advances in many Artificial Intelligence (AI) approaches and techniques. However, while some researchers in this field believe AI is the core element to enhance safety, others believe AI imposes new challenges to assure the safety of these new AI-based systems and applications. In this non-convergent context, this paper presents a systematic literature review to paint a clear picture of the state of the art of the literature in AI on AV safety. Based on an initial sample of 4870 retrieved papers, 59 studies were selected as the result of the selection criteria detailed in the paper. The shortlisted studies were then mapped into six categories to answer the proposed research questions. An AV system model was proposed and applied to orient the discussions about the SLR findings. As a main result, we have reinforced our preliminary observation about the necessity of considering a serious safety agenda for the future studies on AIbased AV systems.

show abstract

Design of Test Platform of Connected-Autonomous Vehicles and Transportation Electrification

Fahad

Othman

2019

Recent Trends in Intelligent Computing, Communication and Devices

View full text Add to dashboard Cite

Observation Based Creation of Minimal Test Suites for Autonomous Vehicles

Cited by 20 publications

References 7 publications

Testing machine learning based systems: a systematic mapping

Testing machine learning based systems: a systematic mapping

A Systematic Literature Review About the Impact of Artificial Intelligence on Autonomous Vehicle Safety

Design of Test Platform of Connected-Autonomous Vehicles and Transportation Electrification

Contact Info

Product

Resources

About