Reinforcement learning for automatic test case prioritization and selection in continuous integration

Spieker, Helge; Gotlieb, Arnaud; Marijan, Dusica; Mossige, Morten

doi:10.1145/3092703.3092709

Cited by 188 publications

(199 citation statements)

References 47 publications

Supporting

Mentioning

195

Contrasting

Order By: Relevance

“…Chen et al [83] proposed a predictive test prioritization technique, which can predict the optimal test prioritization technique for a specific project based on its test distribution information. Spieker et al [84] utilized reinforcement learning to select and prioritize tests based on their duration, previous last execution and failure history. Busjaeger and Xie [85] utilized machine learning to integrate multiple existing test prioritization approaches so as to apply test prioritization in industrial environments.…”

Section: Test Prioritizationmentioning

confidence: 99%

Coverage Prediction for Accelerating Compiler Testing

Chen

Wang

Hao

et al. 2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Compilers are one of the most fundamental software systems. Compiler testing is important for assuring the quality of compilers. Due to the crucial role of compilers, they have to be well tested. Therefore, automated compiler testing techniques (those based on randomly generated programs) tend to run a large number of test programs (which are test inputs of compilers). The cost for compilation and execution for these test programs is significant. These techniques can take a long period of testing time to detect a relatively small number of compiler bugs. That may cause many practical problems, e.g., bringing a lot of costs including time costs and financial costs, and delaying the development/release cycle. Recently, some approaches have been proposed to accelerate compiler testing by executing test programs that are more likely to trigger compiler bugs earlier according to some criteria. However, these approaches ignore an important aspect in compiler testing: different test programs may have similar test capabilities (i.e., testing similar functionalities of a compiler, even detecting the same compiler bug), which may largely discount their acceleration effectiveness if the test programs with similar test capabilities are executed all the time. Test coverage is a proper approximation to help distinguish them, but collecting coverage dynamically is infeasible in compiler testing since most test programs are generated on the fly by automatic test-generation tools like Csmith. In this paper, we propose the first method to predict test coverage statically for compilers, and then propose to prioritize test programs by clustering them according to the predicted coverage information. The novel approach to accelerating compiler testing through coverage prediction is called COP (short for COverage Prediction). Our evaluation on GCC and LLVM demonstrates that COP significantly accelerates compiler testing, achieving an average of 51.01% speedup in test execution time on an existing dataset including three old release versions of the compilers and achieving an average of 68.74% speedup on a new dataset including 12 latest release versions. Moreover, COP outperforms the state-of-the-art acceleration approach significantly by improving 17.16%∼82.51% speedups in different settings on average.

show abstract

Section: Test Prioritizationmentioning

confidence: 99%

Coverage Prediction for Accelerating Compiler Testing

Chen

Wang

Hao

et al. 2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…Our biggest learning was the stick with core fundamental of the design behind an application. The key to success for our system is its strength to determine the workflows and its accuracy in predicting the test cases for any given page based on the relationships learnt based on the historical data [2]. For this particular reason we had to spend time identifying a different approach.…”

Section: Table-vii Data Set Format For Each Pagementioning

confidence: 99%

“…One important driver is the fierce competition from other competing technology companies. Most of the development teams are adopting continuous integration [2] and continuous delivery [7]. Every organization is focused on building better, smarter and reliable software applications with customers in mind.…”

Section: Introductionmentioning

confidence: 99%

Test Case Design and Test Case Prioritization using Machine Learning

Sharna¹,

Agrawal²,

Kumar³

2019

IJEAT

View full text Add to dashboard Cite

Designing and prioritizing test cases is a very tedious task. Given all the advancements in the world of software testing, on any given day engineers spend several man-hours to identify all possible testing scenarios and preconditions attached with it. Test engineers then use the scenarios and preconditions to write multiple test cases. Every test case has a template skeleton to follow - expected results, actual results, priority, test suite category classification (regression, sanity, smoke, integration, etc.), and the respective software (i.e. version, build, release etc.) that has to be tested. Until now there have been efforts to make test case designing simpler by providing software test engineers with tools and processes. But these tools and processes still needs considerable amount of manual intervention in terms of understanding the requirements, analyzing the quality risks and documentation of all possible test scenarios in order to ensure a high quality software delivery. Man-hours spent on test case design and test case prioritization is directly proportional to the cost involved in building software. Our goal was to make sure that the manual intervention in test case design and test case prioritization is reduced to minimum without imposing any software quality risks. So, that the cost to ship and build software is reduced. With this paper we are presenting a solution to this problem. Our goal here was to use machine learning [5] to do automated test case prioritization and creation of test cases for software. In order to achieve this goal we used supervised machine learning [6] approach based on K-Nearest Neighbor classification model for test case design and test case prioritization [4]. On experimenting with other linear and non-linear classifiers we learnt that they did not prove to be as accurate as K-Nearest Neighbor. Our method of machine learning based automated test case design and test case prioritization can be used by any software development organization to reduce it’s software development cost and time taken to ship software to their respective consumers. This aims to benefit the software development industry as a whole.

show abstract

“…The problem of removing redundancy among test cases was addressed in Vangala et al [3] by using unsupervised clustering to cluster similar test cases together. Also, reinforcement learning was used to select and prioritize the test cases [4].…”

Section: Introductionmentioning

confidence: 99%

Machine Learning in Failure Regions Detection and Parameters Analysis

Wahab¹,

Adawi²,

Khater³

2019

IJNDC

View full text Add to dashboard Cite

Testing automation is one of the challenges facing the software development industry, especially for large complex products. This paper proposes a mechanism called Multi Stage Failure Detector (MSFD) for automating black box testing using different machine learning algorithms. The input to MSFD is the tool's set of parameters and their value ranges. The parameter values are randomly sampled to produce a large number of parameter combinations that are fed into the software under test. Using neural networks, the resulting logs from the tool are classified into passing and failing logs and the failing logs are then clustered (using mean-shift clustering) into different failure types. MSFD provides visualization of the failures along with the responsible parameters. Experiments on and results for two real-world complex software products are provided, showing the ability of MSFD to detect all failures and cluster them into the correct failure types, thus reducing the analysis time of failures, improving coverage, and increasing productivity.

show abstract

Reinforcement learning for automatic test case prioritization and selection in continuous integration

Cited by 188 publications

References 47 publications

Coverage Prediction for Accelerating Compiler Testing

Coverage Prediction for Accelerating Compiler Testing

Test Case Design and Test Case Prioritization using Machine Learning

Machine Learning in Failure Regions Detection and Parameters Analysis

Contact Info

Product

Resources

About