Reinforcement Learning for Test Case Prioritization

Bagherzadeh, Mojtaba; Kahani, Nafıseh; Briand, Lionel C.

doi:10.1109/tse.2021.3070549

Cited by 47 publications

(33 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, they failed to use to the state-of-theart ML models in a TCP context (MART) as reported by previous studies [23]. Instead, they trained a point-wise ranking model that, according to existing studies (Bertolino et al [23] and Bagherzadeh et al [6]), provides a lower accuracy compared to pairwise ranking models such as MART. We believe that all these elements explain why the accuracy of their ML models is lower than simple heuristics.…”

Section: Related Workmentioning

confidence: 96%

“…In a similar work to the previous one, Bagherzadeh et al [6] conducted a thorough analysis of ten state-of-the-art deep reinforcement learning techniques in the TCP context. Through an extensive empirical analysis, they showed that the best result from RL techniques can reach a similar accuracy as MART in the TCP context.…”

Section: Related Workmentioning

confidence: 98%

“…We relied on the most recent work [23], [6] to select the best ML model (i.e., MART) for training and evaluating MLbased TCP models based on features. This may threaten the results of our study negatively.…”

Section: External Threatsmentioning

confidence: 99%

“…Collecting coverage information is in general challenging and, furthermore, precise coverage information requires dynamic analysis, which is difficult to apply in practice, more particularly so in a CI context, mainly due to computational overhead and applicability issues [1], [2], [3], [4], [5]. Concerning heuristics based on the execution history, relying solely on such history for TCP may not lead to stable results for complex systems in a CI context [2], [6]. In addition to cost and accuracy issues, heuristics are often defined statically, and there is no standard procedure to tune them based on new changes.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Scalable and Accurate Test Case Prioritization in Continuous Integration Contexts

Yaraghi,

Bagherzadeh,

Kahani

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Continuous Integration (CI) requires efficient regression testing to ensure software quality without significantly delaying its CI builds. This warrants the need for techniques to reduce regression testing time, such as Test Case Prioritization (TCP) techniques that prioritize the execution of test cases to detect faults as early as possible. Many recent TCP studies employ various Machine Learning (ML) techniques to deal with the dynamic and complex nature of CI. However, most of them use a limited number of features for training ML models and evaluate the models on subjects for which the application of TCP makes little practical sense, due to their small regression testing time and low number of failed builds. In this work, we first define, at a conceptual level, a data model that captures data sources and their relations in a typical CI environment. Second, based on this data model, we define a comprehensive set of features that covers all features previously used by related studies. Third, we develop methods and tools to collect the defined features for 25 open-source software systems with enough failed builds and whose regression testing takes at least five minutes. Fourth, relying on the collected dataset containing a comprehensive feature set, we answer four research questions concerning data collection time, the accuracy of ML-based TCP, the impact of the features on accuracy, the decay of ML-based TCP models over time, and the trade-off between data collection time and the accuracy of ML-based TCP techniques.

show abstract

Section: Related Workmentioning

confidence: 96%

Section: Related Workmentioning

confidence: 98%

Section: External Threatsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Scalable and Accurate Test Case Prioritization in Continuous Integration Contexts

Yaraghi,

Bagherzadeh,

Kahani

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The use of ML algorithms for software testing activities has significantly increased in the last few years, 35 covering many activities, including test case generation, 36,37 test case selection, 38 or test prioritization. [38][39][40] Test oracles are one of the core components required to allow full test automation. A recent systematic literature review gathered a total of 22 relevant studies that used ML for the purpose of generating test oracles.…”

Section: Use Of ML For Test Oracle Generationmentioning

confidence: 99%

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Gartziandia

Arrieta

Ayerdi

et al. 2022

J Software Evolu Process

View full text Add to dashboard Cite

The software of systems of elevators needs constant maintenance to deal with new functionality, bug fixes, or legislation changes. To automatically validate the software of these systems, a typical approach in industry is to use regression oracles, which execute test inputs both in the software version under test and in a previous software version. However, these practices require a long test execution time and cannot be re‐used at different test phases. To deal with these issues, we propose Dispatching AlgoRIthm Oracle (DARIO), a test oracle that relies on regression machine‐learning algorithms to detect both functional and non‐functional problems of the system. The machine‐learning algorithms of this oracle are trained by using data from previously tested versions to predict reference functional and non‐functional performance values of the new versions. An empirical evaluation with an industrial case study demonstrates the feasibility of using our approach. A total of five regression learning algorithms were validated by using mutation testing techniques. For the context of functional bugs, the accuracy when predicting verdicts by DARIO ranged between 95% and 98%, across the different scenarios proposed. For the context of non‐functional bugs, were competitive too, having an accuracy when predicting verdicts by DARIO ranged between 83% and 87%.

show abstract

Regression test prioritization leveraging source code similarity with tree kernels

Altiero,

Corazza,

Di Martino

et al. 2024

J Software Evolu Process

View full text Add to dashboard Cite

Regression test prioritization (RTP) is an active research field, aiming at re‐ordering the tests in a test suite to maximize the rate at which faults are detected. A number of RTP strategies have been proposed, leveraging different factors to reorder tests. Some techniques include an analysis of changed source code, to assign higher priority to tests stressing modified parts of the codebase. Still, most of these change‐based solutions focus on simple text‐level comparisons among versions. We believe that measuring source code changes in a more refined way, capable of discriminating between mere textual changes (e.g., renaming of a local variable) and more structural changes (e.g., changes in the control flow), could lead to significant benefits in RTP, under the assumption that major structural changes are also more likely to introduce faults. To this end, we propose two novel RTP techniques that leverage tree kernels (TK), a class of similarity functions largely used in Natural Language Processing on tree‐structured data. In particular, we apply TKs to abstract syntax trees of source code, to more precisely quantify the extent of structural changes in the source code, and prioritize tests accordingly. We assessed the effectiveness of the proposals by conducting an empirical study on five real‐world Java projects, also used in a number of RTP‐related papers. We automatically generated, for each considered pair of software versions (i.e., old version, new version) in the evolution of the involved projects, 100 variations with artificially injected faults, leading to over 5k different software evolution scenarios overall. We compared the proposed prioritization approaches against well‐known prioritization techniques, evaluating both their effectiveness and their execution times. Our findings show that leveraging more refined code change analysis techniques to quantify the extent of changes in source code can lead to relevant improvements in prioritization effectiveness, while typically introducing negligible overheads due to their execution.

show abstract

Reinforcement Learning for Test Case Prioritization

Cited by 47 publications

References 57 publications

Scalable and Accurate Test Case Prioritization in Continuous Integration Contexts

Scalable and Accurate Test Case Prioritization in Continuous Integration Contexts

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Regression test prioritization leveraging source code similarity with tree kernels

Contact Info

Product

Resources

About