FastLane: Test Minimization for Rapidly Deployed Large-Scale Online Services

Philip, Adithya Abraham; Bhagwan, Ranjita; Kumar, Rahul; Maddila, Chandra; Nagappan, Nachiappan

doi:10.1109/icse.2019.00054

Cited by 18 publications

(20 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Effective traditional RTS techniques safely exclude those tests that cannot fail by relying on language-specific white-box program analyses, e.g., recording test-specific execution traces through code instrumentation [26,42,58,65,66,70,79]. However, they are often too costly in large-scale code bases with rapid continuous integration (CI) testing [21,45,62], not capable of collecting test dependencies across language boundaries in multi-language software [12,45,55], and cannot trace third-party libraries [41]. Regression test prioritization (RTP) aims to detect faults earlier by reordering tests through łsurrogatesž [76].…”

Section: Introductionmentioning

confidence: 99%

“…The underlying ranking models exploit different information sources. These include CI test execution logs [3,11,21,71], version control system (VCS) metadata (e.g., number of changed files in commit) [37,62], (textual) differences in code churn [8,49,60,67], and project-or organization-specific information such as static build dependencies [45], flaky test detection signals [45,60,62], or a black-box model of the program inputs [29]. Arguably, access to the latter types of information cannot be guaranteed for arbitrary CI environments.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Empirically evaluating readily available information for regression test optimization in continuous integration

Elsner¹,

Hauer²,

Pretschner³

et al. 2021

Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

Regression test selection (RTS) and prioritization (RTP) techniques aim to reduce testing efforts and developer feedback time after a change to the code base. Using various information sources, including test traces, build dependencies, version control data, and test histories, they have been shown to be effective. However, not all of these sources are guaranteed to be available and accessible for arbitrary continuous integration (CI) environments. In contrast, metadata from version control systems (VCSs) and CI systems are readily available and inexpensive. Yet, corresponding RTP and RTS techniques are scattered across research and often only evaluated on synthetic faults or in a specific industrial context. It is cumbersome for practitioners to identify insights that apply to their context, let alone to calibrate associated parameters for maximum cost-effectiveness. This paper consolidates existing work on RTP and unsafe RTS into an actionable methodology to build and evaluate such approaches that exclusively rely on CI and VCS metadata. To investigate how these approaches from prior research compare in heterogeneous settings, we apply the methodology in a large-scale empirical study on a set of 23 projects covering 37,000 CI logs and 76,000 VCS commits. We find that these approaches significantly outperform established RTP baselines and, while still triggering 90% of the failures, we show that practitioners can expect to save on average 84% of test execution time for unsafe RTS. We also find that it can be beneficial to limit training data, features from test history work better than change-based features, and, somewhat surprisingly, simple and well-known heuristics often outperform complex machine-learned models. CCS CONCEPTS• Software and its engineering → Software testing and debugging.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Empirically evaluating readily available information for regression test optimization in continuous integration

Elsner¹,

Hauer²,

Pretschner³

et al. 2021

Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

show abstract

“…In general, there are two ways to formulate a supervised ML problem for RTP: first, we can formulate it as a binary classification problem, where a failure probability between 0 and 1 is predicted. Similar to prior work, we use three binary classifiers [15], [16], [39]; a logistic regression model (M 1 ), a random forest (M 2 ) and an SVM (M 3 ). Second, we can also directly attempt to predict the failure severity, which can be formulated as a regression problem.…”

Section: P R E P R I N Tmentioning

confidence: 99%

Severity-Aware Prioritization of System-Level Regression Tests in Automotive Software

Wuersching,

Elsner,

Leinen

et al. 2023

2023 IEEE Conference on Software Testing, Verification and Validation (ICST)

View full text Add to dashboard Cite

In automotive software engineering, system-level regression testing is crucial to ensure proper integration of oftentimes safety-critical components. Due to the inherent complexity of such systems and components, testing is commonly performed manually and in a black-box manner, which is particularly costly and leads to slow feedback cycles between testers and developers. Regression Test Prioritization (RTP) aims to reduce feedback time by ordering tests to reveal faults earlier during the testing process. However, most prior RTP research does not incorporate varying fault severity, which must be taken into account when evaluating and designing appropriate RTP approaches for safety-critical automotive software systems. In this work, we present a case study at our industry partner MAN, a leading international provider of commercial vehicles. We design and instantiate a domain-specific, severity-aware RTP assessment model and comparatively assess state-of-the-art RTP approaches. Our results indicate that simple and partly wellknown heuristics based on test history and test costs have the best cost-effectiveness, achieving between 85% and 90% of the maximum possible feedback time reduction. On the other hand, search-based and machine-learning-based RTP approaches do not perform better, especially if available test history is sparse.

show abstract

“…The C++ subtree contains code that compiles into 300+ executable binaries, including 200+ test executables and various applications. Alongside, 700+ binary DLLs are built from the subtree, which are linked against test executables and applications, both, at load-time and run-time 3 .…”

Section: A System Descriptionmentioning

confidence: 99%

BinaryRTS: Cross-language Regression Test Selection for C++ Binaries in CI

Elsner

Kacianka

Lipp

et al. 2023

2023 IEEE Conference on Software Testing, Verification and Validation (ICST)

View full text Add to dashboard Cite

Continuous integration (CI) pipelines are commonly used to execute regression tests before pull requests are merged. Regression test selection (RTS) aims to reduce the required testing effort and feedback time for developers. However, existing RTS techniques are imprecise for tests with cross-language links to compiled C++ binaries or unsafe if tests use external files. This is problematic because modern software in fact involves several programming languages and (non-)code artifacts such as configuration files. In this paper, we present BINARYRTS, a novel RTS technique that leverages dynamic binary instrumentation to collect the covered functions and accessed external files for each test. BINARYRTS then selects tests depending on changes issued to C++ binaries or external (non-)code artifacts. When evaluating BINARYRTS in our large-scale industrial context, we are able to exclude on average up to 74% of tests without missing real failures. We release BINARYRTS as the first publicly available RTS tool for software involving C++ code.

show abstract

FastLane: Test Minimization for Rapidly Deployed Large-Scale Online Services

Cited by 18 publications

References 28 publications

Empirically evaluating readily available information for regression test optimization in continuous integration

Empirically evaluating readily available information for regression test optimization in continuous integration

Severity-Aware Prioritization of System-Level Regression Tests in Automotive Software

BinaryRTS: Cross-language Regression Test Selection for C++ Binaries in CI

Contact Info

Product

Resources

About