Testing machine learning based systems: a systematic mapping

Riccio, Vincenzo; Jahangirova, Gunel; Stocco, Andrea; Humbatova, Nargiz; Weiß, Michael; Tonella, Paolo

doi:10.1007/s10664-020-09881-0

Cited by 144 publications

(61 citation statements)

References 104 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Having checked that no overfitting is present in every fold, for the successive analysis, we used the best trained model across the 10-fold, as reported in Riccio et al (2020) . Such ANN correctly differentiated individuals within ASD and TD groups with an accuracy of 0.93 (sensitivity 0.87, specificity 0.98).…”

Section: Resultsmentioning

confidence: 99%

Using Technology to Identify Children With Autism Through Motor Abnormalities

et al. 2021

View full text Add to dashboard Cite

Autism is a neurodevelopmental disorder typically assessed and diagnosed through observational analysis of behavior. Assessment exclusively based on behavioral observation sessions requires a lot of time for the diagnosis. In recent years, there is a growing need to make assessment processes more motivating and capable to provide objective measures of the disorder. New evidence showed that motor abnormalities may underpin the disorder and provide a computational marker to enhance assessment and diagnostic processes. Thus, a measure of motor patterns could provide a means to assess young children with autism and a new starting point for rehabilitation treatments. In this study, we propose to use a software tool that through a smart tablet device and touch screen sensor technologies could be able to capture detailed information about children’s motor patterns. We compared movement trajectories of autistic children and typically developing children, with the aim to identify autism motor signatures analyzing their coordinates of movements. We used a smart tablet device to record coordinates of dragging movements carried out by 60 children (30 autistic children and 30 typically developing children) during a cognitive task. Machine learning analysis of children’s motor patterns identified autism with 93% accuracy, demonstrating that autism can be computationally identified. The analysis of the features that most affect the prediction reveals and describes the differences between the groups, confirming that motor abnormalities are a core feature of autism.

show abstract

Section: Resultsmentioning

confidence: 99%

Using Technology to Identify Children With Autism Through Motor Abnormalities

et al. 2021

View full text Add to dashboard Cite

show abstract

“…One of the limitations of these approaches is the lack of realism of the generated inputs [44]. While these corrupted images are useful for security testing as adversarial attacks, they are not representative of data captured by sensors of a real DL system.…”

Section: Related Workmentioning

confidence: 99%

Model-based exploration of the frontier of behaviours for deep learning system testing

Riccio

Tonella

2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

Self Cite

View full text Add to dashboard Cite

With the increasing adoption of Deep Learning (DL) for critical tasks, such as autonomous driving, the evaluation of the quality of systems that rely on DL has become crucial. Once trained, DL systems produce an output for any arbitrary numeric vector provided as input, regardless of whether it is within or outside the validity domain of the system under test. Hence, the quality of such systems is determined by the intersection between their validity domain and the regions where their outputs exhibit a misbehaviour. In this paper, we introduce the notion of frontier of behaviours, i.e., the inputs at which the DL system starts to misbehave. If the frontier of misbehaviours is outside the validity domain of the system, the quality check is passed. Otherwise, the inputs at the intersection represent quality deficiencies of the system. We developed DeepJanus, a search-based tool that generates frontier inputs for DL systems. The experimental results obtained for the lane keeping component of a self-driving car show that the frontier of a well trained system contains almost exclusively unrealistic roads that violate the best practices of civil engineering, while the frontier of a poorly trained one includes many valid inputs that point to serious deficiencies of the system. CCS CONCEPTS • Software and its engineering → Software testing and debugging.

show abstract

“…The simulator includes different track circuits and supports training and testing of AVs that performs behavioural cloning, i.e., the AV learns the lane-keeping functionality from a dataset of labeled driving scenes. We selected the Udacity simulator because it is a popular platform used by researchers to evaluate testing techniques for AVs [31,41,43,39,35].…”

Section: Driving Simulatormentioning

confidence: 99%

Quality Metrics and Oracles for Autonomous Vehicles Testing

Jahangirova

Stocco

Tonella

2021

2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST)

Self Cite

View full text Add to dashboard Cite

The race for deploying AI-enabled autonomous vehicles (AVs) on public roads is based on the promise that such self-driving cars will be as safe as or safer than human drivers. Numerous techniques have been proposed to test AVs, which however lack oracle definitions that account for the quality of driving, due to the lack of a commonly used set of metrics. Towards filling this gap, we first performed a systematic analysis of the literature concerning the assessment of the quality of driving of human drivers and extracted 126 metrics. Then, we measured the correlation between such metrics and the human perception of driving quality when AVs are driving. Lastly, we performed a study based on mutation analysis to assess whether the 26 metrics that best capture the quality of AV driving according to the human study can be used as functional oracles. Our results, targeting the Udacity platform, indicate that our automated oracles can kill a high proportion of mutants at a zero or very low false alarm rate, and therefore can be used as effective functional oracles for the quality of driving of AVs.

show abstract

Testing machine learning based systems: a systematic mapping

Cited by 144 publications

References 104 publications

Using Technology to Identify Children With Autism Through Motor Abnormalities

Using Technology to Identify Children With Autism Through Motor Abnormalities

Model-based exploration of the frontier of behaviours for deep learning system testing

Quality Metrics and Oracles for Autonomous Vehicles Testing

Contact Info

Product

Resources

About