Test case selection and prioritization using machine learning: a systematic literature review

Pan, Rongqi; Bagherzadeh, Mojtaba; Ghaleb, Taher Ahmed; Briand, Lionel C.

doi:10.1007/s10664-021-10066-6

Cited by 68 publications

(38 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Compared with the subjects that are used in most recent studies, our work relies on a high number of subjects (25) whose median SLOC is 229k compared to 37.4k [23] and 132k [22], respectively. Also, the average number of test cases per build across our subjects ranges from 33 to 4368, with a median of 117, which is similar to 18 previous studies reported by [8]. Thus, compared to the previous studies, we use a higher number of subjects whose size in terms of SLOC can be considered to be reasonably larger.…”

Section: Subjectssupporting

confidence: 71%

See 1 more Smart Citation

Scalable and Accurate Test Case Prioritization in Continuous Integration Contexts

Yaraghi,

Bagherzadeh,

Kahani

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Continuous Integration (CI) requires efficient regression testing to ensure software quality without significantly delaying its CI builds. This warrants the need for techniques to reduce regression testing time, such as Test Case Prioritization (TCP) techniques that prioritize the execution of test cases to detect faults as early as possible. Many recent TCP studies employ various Machine Learning (ML) techniques to deal with the dynamic and complex nature of CI. However, most of them use a limited number of features for training ML models and evaluate the models on subjects for which the application of TCP makes little practical sense, due to their small regression testing time and low number of failed builds. In this work, we first define, at a conceptual level, a data model that captures data sources and their relations in a typical CI environment. Second, based on this data model, we define a comprehensive set of features that covers all features previously used by related studies. Third, we develop methods and tools to collect the defined features for 25 open-source software systems with enough failed builds and whose regression testing takes at least five minutes. Fourth, relying on the collected dataset containing a comprehensive feature set, we answer four research questions concerning data collection time, the accuracy of ML-based TCP, the impact of the features on accuracy, the decay of ML-based TCP models over time, and the trade-off between data collection time and the accuracy of ML-based TCP techniques.

show abstract

Section: Subjectssupporting

confidence: 71%

“…Thus, many researchers have relied on ML techniques to address TCP in the CI context. According to a recent survey [8], various ML-based TCP techniques have been applied in the CI context. However, existing work has not relied on a comprehensive set of features for training ML models, which is crucial to achieve high accuracy.…”

Section: Introductionmentioning

confidence: 99%

Scalable and Accurate Test Case Prioritization in Continuous Integration Contexts

Yaraghi,

Bagherzadeh,

Kahani

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…A small collection of studies -two out of the four analysed papers deal with RT-was considered to validate the framework. Recently, Pan et al have conducted a systematic literature review of test case selection and prioritisation using ML [84]. As part of their literature analysis, one RQ focused on the types of data features used for training, which are classified into five groups: code complexity, textual data, coverage information, user inputs and history.…”

Section: Other Classifications Of Information Sources For So Ware Tes...mentioning

confidence: 99%

“…A recent trend in TCP is the application of machine learning (ML) [73,84], which has proven to be a suitable alternative to facilitate the automation of different testing tasks [31]. The strength of ML lies in its ability to discover patterns from data and make predictions, so the amount and quality of the input data are critical factors for success [13,56].…”

Section: Introductionmentioning

confidence: 99%

“…One taxonomy also analysed the alignment of testing and requirements activities based on information transfer [110]. Recently, Pan et al have reviewed ML-based TCP literature using four dimensions of classification (techniques, features, evaluation metrics and reproducibility) [84]. As they mention, such dimensions can be viewed as a form of taxonomy to help classify new TCP proposals, although they do not provide a thorough characterisation of the features as we do with respect to information attributes.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Taxonomy of Information Attributes for Test Case Prioritisation: Applicability, Machine Learning

Ramírez,

Feldt,

Romero

2022

Preprint

View full text Add to dashboard Cite

Most software companies have extensive test suites and re-run parts of them continuously to ensure recent changes have no adverse effects. Since test suites are costly to execute, industry needs methods for test case prioritisation (TCP). Recently, TCP methods use machine learning (ML) to exploit the information known about the system under test (SUT) and its test cases. However, the value added by ML-based TCP methods should be critically assessed with respect to the cost of collecting the information. This paper analyses two decades of TCP research, and presents a taxonomy of 91 information attributes that have been used. The attributes are classified with respect to their information sources and the characteristics of their extraction process. Based on this taxonomy, TCP methods validated with industrial data and those applying ML are analysed in terms of information availability, attribute combination and definition of data features suitable for ML. Relying on a high number of information attributes, assuming easy access to SUT code and simplified testing environments are identified as factors that might hamper industrial applicability of ML-based TCP. The T PIA taxonomy provides a reference framework to unify terminology and evaluate alternatives considering the cost-benefit of the information attributes.CCS Concepts: • Software and its engineering → Software testing and debugging; • Computing methodologies → Machine learning.

show abstract

Using goal–question–metric to compare research and practice perspectives on regression testing

Minhas

Koppula

Petersen

et al. 2022

J Software Evolu Process

View full text Add to dashboard Cite

Regression testing is challenging because of its complexity and the amount of effort and time it requires, especially in large-scale environments with continuous integration and delivery. Regression test selection and prioritization techniques have been proposed in the literature to address the regression testing challenges, but adoption rates of these techniques in industry are not encouraging. One of the possible reasons could be the disparity in the regression testing goals in industry and literature. This work compares the research perspective to industry practice on regression testing goals, corresponding information needs, and metrics required to evaluate these goals. We have conducted a literature review of 44 research papers and a survey with 56 testing practitioners. The survey comprises 11 interviews and 45 responses to an online questionnaire. We identified that industry and research accentuate different regression testing goals. For instance, the literature emphasizes increasing the fault detection rates of test suites and early identification of critical faults. In contrast, the practitioners' focus is on test suite maintenance, controlled fault slippage, and awareness of changes. Similarly, the literature suggests maintaining information needs from test case execution histories to evaluate regression testing techniques based on various metrics, whereas, at large, the practitioners do not use the metrics suggested in the literature. To bridge the research and practice gap, based on the literature and survey findings, we have created a goal-question-metric (GQM) model that maps the regression testing goals, associated information needs, and metrics from both perspectives. The GQM model can guide researchers in proposing new techniques closer to industry contexts. Practitioners can benefit from information needs and metrics presented in the literature and can use GQM as a tool to follow their regression testing goals.

show abstract

Test case selection and prioritization using machine learning: a systematic literature review

Cited by 68 publications

References 57 publications

Scalable and Accurate Test Case Prioritization in Continuous Integration Contexts

Scalable and Accurate Test Case Prioritization in Continuous Integration Contexts

A Taxonomy of Information Attributes for Test Case Prioritisation: Applicability, Machine Learning

Using goal–question–metric to compare research and practice perspectives on regression testing

Contact Info

Product

Resources

About