Improve the Quality of ARC Systems Based on the Metamorphic Testing

Zhang, Jihu; Jing, Xiaochuan; Zhang, Wei; Wang, Haipeng; Dong, Yunbai

doi:10.1109/isssr.2016.029

Cited by 2 publications

(9 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An instance of 2018, where the authors provide an example of their test case generation process focusing only on one selected ontology and its conversion into a combinatorial testing input model. Four studies out of 70 (6%) (Abdessalem et al 2016, b, Groce et al 2014 included human evaluation in their empirical approach, while the remaining six (9%) (Zhang et al 2016;Aniculaesei et al 2018;Zhao and Gao 2018;Wolschke et al 2017Wolschke et al , 2018Nakajima 2018) carried out no evaluation of any kind (see Fig. 15).…”

Section: Evaluation Methods (Rq 32)mentioning

confidence: 99%

“…In 12 out of 55 (22%) papers the produced artefact is an oracle (Qin et al 2018;Xie et al 2018;Ding et al 2017;Dwarakanath et al 2018, 2009Nakajima and Bui 2016, 2019Saha and Kanewala 2019;Xie et al 2011;Zhang et al 2016). The main focus of 11 papers from this list is a set of metamorphic relationships (MRs), which are then used to generate a metamorphic oracle.…”

Section: Test Artefacts (Rq 21)mentioning

confidence: 99%

“…Figure 13 provides an overview of the types of oracles that have been adopted with MLSs. The most popular type of oracle is the metamorphic oracle, used in 22 out of 50 (44%) papers (Aniculaesei et al 2018;Ding et al 2017;Du et al 2019;Dwarakanath et al 2018;Guo et al 2018;Murphy et al , b, 2009Nakajima and Bui 2016, 2019Saha and Kanewala 2019;Tian et al 2018;Udeshi et al 2018;Xie et al 2011Xie et al , 2018Xie et al , 2019Zhang et al 2016Zhang et al , 2018bSun et al 2018a;Tuncali et al 2018Tuncali et al , 2019. A central element of a metamorphic oracle is a set of metamorphic relationships that are derived from the innate characteristics of the system under test.…”

Section: ; Tuncalimentioning

confidence: 99%

“…The proposed testing approaches require different levels of access to the MLS under test. In 29 cases out of 70 (41%), it is enough to have a black-box access to the system (Abdessalem et al 2016(Abdessalem et al , 2018aAbeysirigoonawardena et al 2019;Aniculaesei et al 2018;Beglerovic et al 2017;Bolte et al 2019;Bühler and Wegener 2004;Xie et al 2018;Fremont et al 2019;Klueck et al 2018;Li et al 2016;Majumdar et al 2019;Mullins et al 2018;Murphy et al , 2009Nakajima 2018;de Oliveira Neves et al 2016;Patel et al 2018;Qin et al 2018;Rubaiyat et al 2018;Tuncali et al 2018Tuncali et al , 2019Udeshi and Chattopadhyay 2019;Uesato et al 2019;Wolschke et al 2018;Wolschke et al 2017;Zhao and Gao 2018;Zheng et al 2019;Zhang et al 2016). In 11 cases (16%), along with the inputs and outputs of MLS, training data should also be available: black-box access is not sufficient and data-box access to the system should be provided (Cheng et al 2018b;Ding et al 2017;Henriksson et al 2019;Saha and Kanewala 2019;Udeshi et al 2018;Xie et al 2011;Dwarakanath et al 2018;Groce et al 2014;Zhang et al 2018b;Murphy et al 2007b;…”

Section: Access To the System (Rq 25)mentioning

confidence: 99%

“…Three studies (Bühler and Wegener 2004;Byun et al 2019;Zheng et al 2019) perform an evaluation on proprietary systems belonging to an industrial partner. Five works (Murphy et al 2007a, Abdessalem et al 2016) adopted a combined approach, featuring an evaluation in both academic and industrial contexts, while six works (Zhang et al 2016;Aniculaesei et al 2018;Zhao and Gao 2018;Wolschke et al 2017, Nakajima 2018 contained no empirical evaluation of any sort. Taking into account that ML is widely used in industry and its applications are growing exponentially, the collected data suggest that industrial research should be given greater focus.…”

Section: Evaluation Type (Rq 31)mentioning

confidence: 99%

See 4 more Smart Citations

Testing machine learning based systems: a systematic mapping

et al. 2020

View full text Add to dashboard Cite

Context: A Machine Learning based System (MLS) is a software system including one or more components that learn how to perform a task from a given data set. The increasing adoption of MLSs in safety critical domains such as autonomous driving, healthcare, and finance has fostered much attention towards the quality assurance of such systems. Despite the advances in software testing, MLSs bring novel and unprecedented challenges, since their behaviour is defined jointly by the code that implements them and the data used for training them. Objective: To identify the existing solutions for functional testing of MLSs, and classify them from three different perspectives: (1) the context of the problem they address, (2) their features, and (3) their empirical evaluation. To report demographic information about the ongoing research. To identify open challenges for future research. Method: We conducted a systematic mapping study about testing techniques for MLSs driven by 33 research questions. We followed existing guidelines when defining our research protocol so as to increase the repeatability and reliability of our results. Results: We identified 70 relevant primary studies, mostly published in the last years. We identified 11 problems addressed in the literature. We investigated multiple aspects of the testing approaches, such as the used/proposed adequacy criteria, the algorithms for test input generation, and the test oracles. Conclusions: The most active research areas in MLS testing address automated scenario/input generation and test oracle creation. MLS testing is a rapidly growing and developing research area, with many open challenges, such as the generation of realistic inputs and the definition of reliable evaluation metrics and benchmarks.

show abstract