Coverage-Based Reduction of Test Execution Time: Lessons from a Very Large Industrial Project

Bach, Thomas; Andrzejak, Artur; Pannemans, Ralf

doi:10.1109/icstw.2017.6

Cited by 12 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our solution acts in a similar way in Algorithm 3 as we order the test runs based on the similarity of the test configurations they involve, however our solution does not only reduce the execution time but also the disturbance based on the level of criticality of the configured instances impacted by the testing. Bach et al (2017) also proposes a method to reduce the time needed to achieve certain test coverage through test case selection. The main idea proposed by the authors consists of using overlap-aware test case selection algorithms in order to reduce the amount of coverage items covered by different test cases which helps reduce the test execution time.…”

Section: Related Workmentioning

confidence: 99%

Test Plan Generation for Live Testing of Cloud Services

Jebbar¹,

Khendek²,

Toeroe

2023

Preprint

View full text Add to dashboard Cite

Live testing is performed in the production environment ideally without causing unacceptable disturbance to the production traffic. Thus, test activities have to be orchestrated properly to avoid test interferences with the production traffic. A test plan is the road map that specifies how the test activities need to be orchestrated. Developing a test plan includes tasks such as test configuration selection/generation, test configuration deployment planning, creating the test runs schedule, choosing strategies to mitigate the risk of test interferences, etc. The manual design of a test plan is tedious and error prone. This task becomes harder especially when the systems are large and complex. In this paper we propose an approach for automating test plans generation. With this approach we aim at reducing the time needed to execute the test plan while maintaining the service disruption at an acceptable level. We illustrate our approach with a case study and discuss its different aspects.

show abstract

Section: Related Workmentioning

confidence: 99%

Test Plan Generation for Live Testing of Cloud Services

Jebbar¹,

Khendek²,

Toeroe

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Flaky tests have been reported to be a significant problem in practice at many companies including Google, Huawei, Microsoft, SAP, Spotify, Mozilla, and Facebook [12][13][14][15]. As reported by Google, almost 16% of their 4.2 million test cases are flaky.…”

Section: Flaky Testsmentioning

confidence: 99%

Flakify: A Black-Box, Language Model-based Predictor for Flaky Tests

Fatima,

Ghaleb,

Briand

2021

Preprint

View full text Add to dashboard Cite

Software testing assures that code changes do not adversely affect existing functionality. However, a test case can be flaky, i.e., passing and failing across executions, even for the same version of the source code. Flaky tests introduce overhead to software development as they can lead to unnecessary attempts to debug production or testing code. Besides rerunning test cases multiple times, which is time-consuming and computationally expensive, flaky tests can be predicted using machine learning (ML) models. However, the state-of-the-art ML-based flaky test predictors rely on pre-defined sets of features that are either project-specific, i.e., inapplicable to other projects, or require access to production code, which is not always available to software test engineers. Moreover, given the non-deterministic behavior of flaky tests, it can be challenging to determine a complete set of features that could potentially be associated with test flakiness. Therefore, in this paper, we propose Flakify, a black-box, language model-based predictor for flaky tests. Flakify does not require to (a) rerun test cases, (b) pre-define features, or (c) access to production code. To this end, we employ CodeBERT, a pre-trained language model, and fine-tune it to predict flaky tests by relying exclusively on the source code of test cases. We evaluated Flakify on a publicly available dataset and compared our results with FlakeFlagger, the best state-of-the-art ML-based, white-box predictor for flaky tests. Flakify surpasses FlakeFlagger by 10 and 18 percentage points (pp) in terms of precision and recall, respectively, thus reducing the effort bound to be wasted on unnecessarily debugging test cases and production code by the same percentages (corresponding to a reduction rate of 25% and 64%), respectively. Our results further show that a black-box version of FlakeFlagger is not a viable option for predicting flaky tests.

show abstract

“…Therefore, SAP maintains an extensive set of over 100 000 regression tests, i.e. tests that assure a new software version has no regressions [3], [4]. However, even with such an extensive test suite, customers may use the software in a way that is not explicitly tested.…”

Section: Introductionmentioning

confidence: 99%

Automatic Error Classification and Root Cause Determination while Replaying Recorded Workload Data at SAP HANA

Jambigi¹,

Bach²,

Schabernack³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Capturing customer workloads of database systems to replay these workloads during internal testing can be beneficial for software quality assurance. However, we experienced that such replays can produce a large amount of false positive alerts that make the results unreliable or time consuming to analyze. Therefore, we design a machine learning based approach that attributes root causes to the alerts. This provides several benefits for quality assurance and allows for example to classify whether an alert is true positive or false positive. Our approach considerably reduces manual effort and improves the overall quality assurance for the database system SAP HANA. We discuss the problem, the design and result of our approach, and we present practical limitations that may require further research.

show abstract

Coverage-Based Reduction of Test Execution Time: Lessons from a Very Large Industrial Project

Cited by 12 publications

References 21 publications

Test Plan Generation for Live Testing of Cloud Services

Test Plan Generation for Live Testing of Cloud Services

Flakify: A Black-Box, Language Model-based Predictor for Flaky Tests

Automatic Error Classification and Root Cause Determination while Replaying Recorded Workload Data at SAP HANA

Contact Info

Product

Resources

About