Supervised learning over test executions as a test oracle

Tsimpourlas, Foivos; Rajan, Ajitha; Allamanis, Miltiadis

doi:10.1145/3412841.3442027

Cited by 9 publications

(10 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We have implemented an end-to-end framework for automating the steps in our approach, (1) gathering execution traces as sequences of method invocations, (2) encoding variable length execution traces into a fixed length vector, and (3) designing a NN model that uses the trace information to classify the trace as pass or fail. We augmented our work in [7] by supporting Java programmes in addition to C/C++ in Step 1. In addition, we conducted an extensive evaluation using 15 realistic PUTs and tests.…”

Section: Discussionmentioning

confidence: 99%

“…Other bodies of work in programme analysis have used NNs to predict method or variable names and detect name‐based bug patterns [10, 11] relying on static programme information, namely, embeddings of the Abstract Syntax Tree or source code. Our approach in [7] is the first attempt at using dynamic execution trace information in NN models for classifying test executions and has the following steps: Instrument a programme to gather execution traces as sequences of method invocations. Label a small fraction of the traces with their classification decision. Design a NN component that embeds the execution traces to fixed length vectors. Design a NN component that uses the line‐by‐line trace information to classify traces as pass or fail. Train a NN model that combines the above components and evaluate it on unseen execution traces for that programme. …”

Section: Introductionmentioning

confidence: 99%

“…The contributions of this study, different from our previous study, are summarised as follows: Support for Java programs. Our work in [7] provided tool support in the low level virtual machine (LLVM) [12] framework to instrument the intermediate representation (LLVM‐IR) of programmes to gather execution traces. LLVM, however, does not provide front‐end support for Java programmes.…”

Section: Introductionmentioning

confidence: 99%

“… Extensive empirical evaluation. We augment the experiments in [7] with 10 additional subject programmes—9 network protocols from L7‐filter [14] and 1 Java utilities library from Defects4J [15], a database of real faults for open‐source Java programmes. For these subject programmes, we evaluate the precision, recall and specificity of our approach in classifying execution traces.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Embedding and classifying test execution traces using neural networks

et al. 2021

Self Cite

View full text Add to dashboard Cite

Classifying test executions automatically as pass or fail remains a key challenge in software testing and is referred to as the test oracle problem. It is being attempted to solve this problem with supervised learning over test execution traces. A programme is instrumented to gather execution traces as sequences of method invocations. A small fraction of the programme's execution traces is labelled with pass or fail verdicts. Execution traces are then embedded as fixed length vectors and a neural network (NN) component that uses the line-by-line information to classify traces as pass or fail is designed. The classification accuracy of this approach is evaluated using subject programs from different application domains-1. Module from Ethereum Blockchain, 2. Module from PyTorch deep learning framework, 3. Microsoft SEAL encryption library components, 4. Sed stream editor, 5. Nine network protocols from Linux packet identifier, L7-Filter and 6. Utilities library, commons-lang for Java. For all subject programs, it was found that test execution classification had high precision, recall and specificity, averaging to 93%, 94% and 96%, respectively, while only training with an average 14% of the total traces. Experiments show that the proposed NN-based approach is promising in classifying test executions from different application domains. K E Y W O R D Sexecution trace, neural networks, software testing, test oracle CCS CONCEPTS • Software and its engineering → Software testing and debugging; • Computing methodologies → Supervised learning by classification.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Embedding and classifying test execution traces using neural networks

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…Test Verdicts: [80,81,77,82] employ various NNs to train a model that predicts verdicts. [82] train models for complex programs using a deep NN with long-short term memory (LSTM). [75] use adaptive boosting, an ensemble technique.…”

Section: Test Oracle Generationmentioning

confidence: 99%

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Fontes¹,

Gay²

2022

Preprint

View full text Add to dashboard Cite

Context: Machine learning (ML) may enable effective automated test generation.Objectives: We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges.Methods: We perform a systematic literature review on a sample of 97 publications.Results: ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property-based, and expected output oracles. Supervised learning-often based on neural networks-and reinforcement learning-often based on Q-learning-are common, and some publications also employ unsupervised or semi-supervised learning. (Semi-/Un-)Supervised approaches are evaluated using both traditional testing metrics and ML-related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. Conclusion:Work-to-date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed-and how they are applied-benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.

show abstract

The integration of machine learning into automated test generation: A systematic mapping study

Fontes

Gay

2023

Software Testing Verif & Rel

View full text Add to dashboard Cite

Machine learning (ML) may enable effective automated test generation. We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges in this intersection by performing. We perform a systematic mapping study on a sample of 124 publications. ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property‐based, and expected output oracles. Supervised learning—often based on neural networks—and reinforcement learning—often based on Q‐learning—are common, and some publications also employ unsupervised or semi‐supervised learning. (Semi‐/Un‐)Supervised approaches are evaluated using both traditional testing metrics and ML‐related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. The work‐to‐date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed—and how they are applied—benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.

show abstract

Supervised learning over test executions as a test oracle

Cited by 9 publications

References 31 publications

Embedding and classifying test execution traces using neural networks

Embedding and classifying test execution traces using neural networks

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

The integration of machine learning into automated test generation: A systematic mapping study

Contact Info

Product

Resources

About