Test Case Prioritization in Continuous Integration environments: A systematic mapping study

Lima, Jackson Prado; Vergílio, Silvia Regina

doi:10.1016/j.infsof.2020.106268

Cited by 65 publications

(44 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, to review the use of TCP techniques in web services, a study [32] has identified statistical methods, metrics to validate the proposed technique, and issues relating to current TCP concerning web services. Furthermore, a study [33] reviewed the scope of TCP's application for continuous interaction (TCPCI) environment. Some important aspects were analysed, including problems in continuous integration (CI), sources of information (input resources) for TCP in TCPCI, evaluating measures using metrics in TCP, and analysis of research opportunities to guide future research.…”

Section: B Related Workmentioning

confidence: 99%

A Systematic Literature Review on Regression Test Case Prioritization

Rahmani¹,

Ahmad²,

Jalil³

et al. 2021

IJACSA

View full text Add to dashboard Cite

Test case prioritization (TCP) is deemed valid to improve testing efficiency, especially in regression testing, as retest all is costly. The TCP schedule the test case execution order to detect bugs faster. For such benefit, test case prioritization has been intensively studied. This paper reviews the development of TCP for regression testing with 48 papers from 2017 to 2020. In this paper, we present four critical surveys. First is the development of approaches and techniques in regression TCP studies, second is the identification of software under test (SUT) variations used in TCP studies, third is the trend of metrics used to measure the TCP studies effectiveness, and fourth is the stateof-the-art of requirements-based TCP. Furthermore, we discuss development opportunities and potential future directions on regression TCP. Our review provides evidence that TCP has increasing interests. We also discovered that requirement-based utilization would help to prepare test cases earlier to improve TCP effectiveness.

show abstract

Section: B Related Workmentioning

confidence: 99%

A Systematic Literature Review on Regression Test Case Prioritization

Rahmani¹,

Ahmad²,

Jalil³

et al. 2021

IJACSA

View full text Add to dashboard Cite

show abstract

“…Regarding TCP in CI environments we refer a recent mapping [30] that highlights some works for the HCS context [19,20,21,22]. The approach of Marijan et al [20,21] uses historical test data to determine an optimal order of test cases to ensure feature coverage, early fault detection and execution time.…”

Section: Related Workmentioning

confidence: 99%

“…A recent mapping reports approaches that adapt TCP techniques for CI environments [30], some of them addressing the HCS context [19,21,22]. But most existing approaches present some limitations.…”

Section: Introductionmentioning

confidence: 99%

Learning-based prioritization of test cases in continuous integration of highly-configurable software

Lima

Mendonça

Vergílio

et al. 2020

Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume a - Volume A

Self Cite

View full text Add to dashboard Cite

Continuous Integration (CI) is a practice widely adopted in the industry to allow frequent integration of code changes. During the CI process, many test cases are executed multiple times a day, subject to time constraints. In this scenario, a learning-based approach, named COLEMAN, has been successfully applied. COLEMAN allows earlier execution of the most promising test cases to reveal faults. This approach considers CI particularities such as time budget and volatility of test cases, related to the fact that test cases can be added/removed along the CI cycles. In the CI of Highly Configuration System (HCS), many product variants must be tested, each one with different configuration options, but having test cases that are common to or reused from other variants. In this context, we found, by analogy, another particularity, the volatility of variants, that is, some variants can be included/discontinued along CI cycles. Considering this context, this work introduces two strategies for the application of COLEMAN in the CI of HCS: the Variant Test Set Strategy (VTS) that relies on the test set specific for each variant, and the Whole Test Set Strategy (WST) that prioritizes the test set composed by the union of the test cases of all variants. Both strategies are evaluated in a real-world HCS, considering three test budgets. The results show that the proposed strategies are applicable regarding the time spent for prioritization. They perform similarly regarding early fault detection, but WTS better mitigates the problem of beginning without knowledge, and is more suitable when a new variant to be tested is added. CCS CONCEPTS • General and reference → Experimentation; • Software and its engineering → Software product lines; Software testing and debugging.

show abstract

“…The developed approach leverages dynamic relation-based coverage as a measurement to extend the original additional greedy coverage procedure in TCP techniques. Lima and Vergilio [38] presented the outcomes of a systematic mapping investigation on TCP in continuous integration (TCPCI) settings that reported the key features of TCPCI models and their assessment facets. The mapping was used as a plan that contains the definition of RQs, assessment attributes, search string, and evaluation of search engines.…”

Section: Literature Review On Tcpmentioning

confidence: 99%

An Integrated Decision Approach with Probabilistic Linguistic Information for Test Case Prioritization

et al. 2020

View full text Add to dashboard Cite

This paper focuses on an exciting and essential problem in software companies. The software life cycle includes testing software, which is often time-consuming, and is a critical phase in the software development process. To reduce time spent on testing and to maintain software quality, the idea of a systematic selection of test cases is needed. Attracted by the claim, researchers presented test case prioritization (TCP) by applying the concepts of multi-criteria decision-making (MCDM). However, the literature on TCP suffers from the following issues: (i) difficulty in properly handling uncertainty; (ii) systematic evaluation of criteria by understanding the hesitation of experts; and (iii) rational prioritization of test cases by considering the nature of criteria. Motivated by these issues, an integrated approach is put forward that could circumvent the problem in this paper. The main aim of this research is to develop a decision model with integrated methods for TCP. The core importance of the proposed model is to (i) provide a systematic/methodical decision on TCP with a reduction in testing time and cost; (ii) help software personnel choose an apt test case from the suite for testing software; (iii) reduce human bias by mitigating intervention of personnel in the decision process. To this end, probabilistic linguistic information (PLI) is adopted as the preference structure that could flexibly handle uncertainty by associating occurrence probability to each linguistic term. Furthermore, an attitude-based entropy measure is presented for criteria weight calculation, and finally, the EDAS ranking method is extended to PLI for TCP. An empirical study of TCP in a software company is presented to certify the integrated approach’s effectiveness. The strengths and weaknesses of the introduced approach are conferred by comparing it with the relevant methods.

show abstract

Test Case Prioritization in Continuous Integration environments: A systematic mapping study

Cited by 65 publications

References 22 publications

A Systematic Literature Review on Regression Test Case Prioritization

A Systematic Literature Review on Regression Test Case Prioritization

Learning-based prioritization of test cases in continuous integration of highly-configurable software

An Integrated Decision Approach with Probabilistic Linguistic Information for Test Case Prioritization

Contact Info

Product

Resources

About