Using machine learning to generate test oracles: a systematic literature review

Fontes, Afonso

doi:10.1145/3472675.3473974

Cited by 11 publications

(20 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…14 These approaches construct models by observing data using statistical analyses, where learning begins with the search for patterns inside the data used for training the algorithm. 15 ML can be categorized in the categories of supervised, unsupervised, and reinforcement learning. 16 In supervised learning, the training data consist of some input vectors with their respective labels, which indicate the output expected for each input vector.…”

Section: Algorithmsmentioning

confidence: 99%

“…15 Lastly, reinforcement learning approaches aim at selecting actions with certain goals. 15 They use feedback on the effect of the action taken (i.e., rewards), which is later used to improve the estimation and iteratively perform more effective actions. 15 In our study, we use supervised learning algorithms aimed at solving a regression problem.…”

Section: Algorithmsmentioning

confidence: 99%

See 1 more Smart Citation

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Gartziandia

Arrieta

Ayerdi

et al. 2022

J Software Evolu Process

View full text Add to dashboard Cite

The software of systems of elevators needs constant maintenance to deal with new functionality, bug fixes, or legislation changes. To automatically validate the software of these systems, a typical approach in industry is to use regression oracles, which execute test inputs both in the software version under test and in a previous software version. However, these practices require a long test execution time and cannot be re‐used at different test phases. To deal with these issues, we propose Dispatching AlgoRIthm Oracle (DARIO), a test oracle that relies on regression machine‐learning algorithms to detect both functional and non‐functional problems of the system. The machine‐learning algorithms of this oracle are trained by using data from previously tested versions to predict reference functional and non‐functional performance values of the new versions. An empirical evaluation with an industrial case study demonstrates the feasibility of using our approach. A total of five regression learning algorithms were validated by using mutation testing techniques. For the context of functional bugs, the accuracy when predicting verdicts by DARIO ranged between 95% and 98%, across the different scenarios proposed. For the context of non‐functional bugs, were competitive too, having an accuracy when predicting verdicts by DARIO ranged between 83% and 87%.

show abstract

Section: Algorithmsmentioning

confidence: 99%

Section: Algorithmsmentioning

confidence: 99%

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Gartziandia

Arrieta

Ayerdi

et al. 2022

J Software Evolu Process

View full text Add to dashboard Cite

show abstract

“…Oracle generation has long been seen as a major challenge for test automation research [1,2]. However, ML is a realistic route to achieve automated oracle generation [6], and publications have started to appear (33%).…”

Section: Rq1: Testing Practices Addressedmentioning

confidence: 99%

“…This study extends an initial SLR on test oracle generation[6]. Our extended study also includes input generation, updates the sample of publications, and features an extended analysis and discussion.…”

mentioning

confidence: 93%

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Fontes¹,

Gay²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Context: Machine learning (ML) may enable effective automated test generation.Objectives: We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges.Methods: We perform a systematic literature review on a sample of 97 publications.Results: ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property-based, and expected output oracles. Supervised learning-often based on neural networks-and reinforcement learning-often based on Q-learning-are common, and some publications also employ unsupervised or semi-supervised learning. (Semi-/Un-)Supervised approaches are evaluated using both traditional testing metrics and ML-related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. Conclusion:Work-to-date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed-and how they are applied-benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.

show abstract

“…An overview of attempts to use machine learning to derive oracles is offered by Fontes and Gay:[22].…”

mentioning

confidence: 99%

Automated Support for Unit Test Generation: A Tutorial Book Chapter

Fontes¹,

Gay²,

Neto³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Unit testing is a stage of testing where the smallest segment of code that can be tested in isolation from the rest of the system-often a class-is tested. Unit tests are typically written as executable code, often in a format provided by a unit testing framework such as pytest for Python.Creating unit tests is a time and effort-intensive process with many repetitive, manual elements. To illustrate how AI can support unit testing, this chapter introduces the concept of search-based unit test generation. This technique frames the selection of test input as an optimization problem-we seek a set of test cases that meet some measurable goal of a tester-and unleashes powerful metaheuristic search algorithms to identify the best possible test cases within a restricted timeframe. This chapter introduces two algorithms that can generate pytest-formatted unit tests, tuned towards coverage of source code statements. The chapter concludes by discussing more advanced concepts and gives pointers to further reading for how artificial intelligence can support developers and testers when unit testing software.

show abstract

Using machine learning to generate test oracles: a systematic literature review

Cited by 11 publications

References 36 publications

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Automated Support for Unit Test Generation: A Tutorial Book Chapter

Contact Info

Product

Resources

About