Build system aware multi-language regression test selection in continuous integration

Elsner, Daniel; Wuersching, Roland; Schnappinger, Markus; Pretschner, Alexander; Graber, Maria; Dammer, René; Reimer, Silke

doi:10.1145/3510457.3513078

Cited by 7 publications

(33 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most RTP studies target unit-level testing whereas this study focuses on system-level regression testing. We have motivated why RTP approaches that merely use black-box metadata are often more suitable for system-level regression testing, given that static and dynamic program analysis results are either not available, limited to a single system or programming language, or may induce prohibitive overhead in production deployments [16], [21], [25]- [27]. This study aims to compare existing techniques and develop new approaches for RTP in system-level regression testing in an industrial context, more specifically for embedded automotive software at MAN.…”

Section: Prioritizing System-level Regression Test Casesmentioning

confidence: 99%

Severity-Aware Prioritization of System-Level Regression Tests in Automotive Software

Wuersching,

Elsner,

Leinen

et al. 2023

2023 IEEE Conference on Software Testing, Verification and Validation (ICST)

Self Cite

View full text Add to dashboard Cite

In automotive software engineering, system-level regression testing is crucial to ensure proper integration of oftentimes safety-critical components. Due to the inherent complexity of such systems and components, testing is commonly performed manually and in a black-box manner, which is particularly costly and leads to slow feedback cycles between testers and developers. Regression Test Prioritization (RTP) aims to reduce feedback time by ordering tests to reveal faults earlier during the testing process. However, most prior RTP research does not incorporate varying fault severity, which must be taken into account when evaluating and designing appropriate RTP approaches for safety-critical automotive software systems. In this work, we present a case study at our industry partner MAN, a leading international provider of commercial vehicles. We design and instantiate a domain-specific, severity-aware RTP assessment model and comparatively assess state-of-the-art RTP approaches. Our results indicate that simple and partly wellknown heuristics based on test history and test costs have the best cost-effectiveness, achieving between 85% and 90% of the maximum possible feedback time reduction. On the other hand, search-based and machine-learning-based RTP approaches do not perform better, especially if available test history is sparse.

show abstract

Section: Prioritizing System-level Regression Test Casesmentioning

confidence: 99%

Severity-Aware Prioritization of System-Level Regression Tests in Automotive Software

Wuersching,

Elsner,

Leinen

et al. 2023

2023 IEEE Conference on Software Testing, Verification and Validation (ICST)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Since crosslanguage links to XML are often employed to configure DI frameworks [21], projects using DI are particularly prone to these violations. Shi et al [18] and Elsner et al [12] further report (potential) safety violations related to missing edges in static and dynamic dependency graphs, respectively, which are introduced through DI frameworks' source code annotations. When for example the injection priority of beans is changed, this may affect the run-time behavior of a test without modifying any of the entities covered by the test before the change [12].…”

Section: B Di-related Safety Violations In Regression Test Selectionmentioning

confidence: 99%

“…Shi et al [18] and Elsner et al [12] further report (potential) safety violations related to missing edges in static and dynamic dependency graphs, respectively, which are introduced through DI frameworks' source code annotations. When for example the injection priority of beans is changed, this may affect the run-time behavior of a test without modifying any of the entities covered by the test before the change [12]. For the rest of the paper, we collectively refer to changes related to DI mechanics as DI-related changes.…”

Section: B Di-related Safety Violations In Regression Test Selectionmentioning

confidence: 99%

“…However, Zhu et al [17] discovered that this assumption may not hold true in practice: using their tool RTSCHECK, they found safety violations in state-of-the-art RTS tools due to changes to non-Java (source) files (e.g., XML files). More recently, Shi et al [18] as well as Elsner et al [12] identified cases where RTS tools are not able to recognize Java or XML code changes which may alter the run-time behavior of tests through dependency injection (DI), where objects are dynamically created and injected. Although DI is a widely adopted design principle to reduce coupling in object-oriented software-it is heavily used in Java EE and Spring [19], [20]contemporary RTS tools lack adequate DI support.…”

Section: Introductionmentioning

confidence: 99%

“…We find that in 33.3% of the analyzed commits DI-related P r e p r i n t changes affect some tests, whereas STARTS misses to select affected tests in 3.1% (7) commits. Despite its superior safety for DI-related changes, DIRTS has comparable efficiency and precision and, contrary to STARTS, does not require a fully compiled Java workspace, giving more flexibility for selective build optimization [12].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

DIRTS: Dependency Injection Aware Regression Test Selection

Hundsdorfer,

Elsner,

Pretschner

2023

2023 IEEE Conference on Software Testing, Verification and Validation (ICST)

Self Cite

View full text Add to dashboard Cite

Regression test selection (RTS) aims to reduce regression testing effort by selecting only those tests that are affected by introduced changes. RTS techniques are considered to be safe if they select all affected test cases. Several supposedly safe RTS tools have been developed over the past decades, lately especially for Java projects. However, recent studies have shown that state-of-the-art RTS tools for Java can become unsafe when confronted with dependency injection (DI) mechanisms: despite the widespread use of DI frameworks in Java projects, no existing technique acknowledges DI-related changes. In this paper, we analyze the reasons behind unsafe RTS behavior for DI-related changes and develop DIRTS, a novel DI-aware RTS tool for Java. To counteract effects of DI on RTS, DIRTS efficiently analyzes source code annotations and metadata employed by popular DI frameworks, and generates a dependency graph including edges for dynamically injected objects. We evaluate DIRTS on 228 commits from 9 open-source Java projects that use DI. Our results indicate that in 33.3% of those commits DI-related changes affect some tests, and in 3.1% (7) DIRTS identifies affected tests that are clearly missed by the static RTS tool STARTS. Still, DIRTS is comparatively efficient and precise. We publish DIRTS 1,2 as an RTS tool that can either be used as a safety extension for existing RTS tools or as a standalone RTS solution.

show abstract

BinaryRTS: Cross-language Regression Test Selection for C++ Binaries in CI

Elsner

Kacianka

Lipp

et al. 2023

2023 IEEE Conference on Software Testing, Verification and Validation (ICST)

Self Cite

View full text Add to dashboard Cite

Continuous integration (CI) pipelines are commonly used to execute regression tests before pull requests are merged. Regression test selection (RTS) aims to reduce the required testing effort and feedback time for developers. However, existing RTS techniques are imprecise for tests with cross-language links to compiled C++ binaries or unsafe if tests use external files. This is problematic because modern software in fact involves several programming languages and (non-)code artifacts such as configuration files. In this paper, we present BINARYRTS, a novel RTS technique that leverages dynamic binary instrumentation to collect the covered functions and accessed external files for each test. BINARYRTS then selects tests depending on changes issued to C++ binaries or external (non-)code artifacts. When evaluating BINARYRTS in our large-scale industrial context, we are able to exclude on average up to 74% of tests without missing real failures. We release BINARYRTS as the first publicly available RTS tool for software involving C++ code.

show abstract

Build system aware multi-language regression test selection in continuous integration

Cited by 7 publications

References 27 publications

Severity-Aware Prioritization of System-Level Regression Tests in Automotive Software

Severity-Aware Prioritization of System-Level Regression Tests in Automotive Software

DIRTS: Dependency Injection Aware Regression Test Selection

BinaryRTS: Cross-language Regression Test Selection for C++ Binaries in CI

Contact Info

Product

Resources

About