Evaluating Regression Test Selection Opportunities in a Very Large Open-Source Ecosystem

Gyori, Alex; Legunsen, Owolabi; Hariri, Farah; Marinov, Darko

doi:10.1109/issre.2018.00022

Cited by 23 publications

(10 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Class-level RTS was determined to be more practical and time-saving than method-level RTS. Gyori et al [31] compared variants of dynamic and static class-level RTS with project-level RTS in the Maven Central open-source ecosystem. Classlevel RTS was found to be an order of magnitude less costly than project-level RTS in terms of test selection reduction.…”

Section: Related Workmentioning

confidence: 99%

Bridging the model-to-code abstraction gap with fuzzy logic in model-based regression test selection

et al. 2021

View full text Add to dashboard Cite

Regression test selection (RTS) approaches reduce the cost of regression testing of evolving software systems. Existing RTS approaches based on UML models use behavioral diagrams or a combination of structural and behavioral diagrams. However, in practice, behavioral diagrams are incomplete or not used. In previous work, we proposed a fuzzy logic based RTS approach called FLiRTS that uses UML sequence and activity diagrams. In this work, we introduce FLiRTS 2, which drops the need for behavioral diagrams and relies on system models that only use UML class diagrams, which are the most widely used UML diagrams in practice. FLiRTS 2 addresses the unavailability of behavioral diagrams by classifying test cases using fuzzy logic after analyzing the information commonly provided in class diagrams. We evaluated FLiRTS 2 on UML class diagrams extracted from 3331 revisions of 13 open-source software systems, and compared the results with those of code-based dynamic (Ekstazi) and static (STARTS) RTS approaches. The average test suite reduction using FLiRTS 2 was 82.06%. The average safety violations of FLiRTS 2 with respect to Ekstazi and STARTS were 18.88% and 16.53%, respectively. FLiRTS 2 selected on average about 82% of the test cases that were selected by Ekstazi and STARTS. The average precision violations of FLiRTS 2 with respect to Ekstazi and STARTS were 13.27% and 9.01%, respectively. The average mutation score of the full test suites was 18.90%; the standard deviation of the reduced test suites from the average deviation of the mutation score for each subject was 1.78% for FLiRTS 2, 1.11% for Ekstazi, and 1.43% for STARTS. Our experiment demonstrated that the performance of FLiRTS 2 is close to the state-of-art tools for code-based RTS but requires less information and performs the selection in less time.

show abstract

Section: Related Workmentioning

confidence: 99%

Bridging the model-to-code abstraction gap with fuzzy logic in model-based regression test selection

et al. 2021

View full text Add to dashboard Cite

show abstract

“…We ] also used STARTS for change-impact analysis when adapting runtime verification to the context of software evolution by focusing runtime verification and its users on affected parts of code. We compared the class-level RTS in STARTS with the module-level RTS commonly used in large software ecosystems, e.g., at Facebook, Google and Microsoft [Gyori et al 2018]. We found that STARTS can reduce the test-selection rate in those ecosystems by 10x.…”

Section: Related Workmentioning

confidence: 99%

Reflection-aware static regression test selection

Shi

Hadzi-Tanovic

Zhang³

et al. 2019

Proc. ACM Program. Lang.

Self Cite

View full text Add to dashboard Cite

Regression test selection (RTS) aims to speed up regression testing by rerunning only tests that are affected by code changes. RTS can be performed using static or dynamic analysis techniques. Our prior study showed that static and dynamic RTS perform similarly for medium-sized Java projects. However, the results of that prior study also showed that static RTS can be unsafe, missing to select tests that dynamic RTS selects, and that reflection was the only cause of unsafety observed among the evaluated projects. In this paper, we investigate five techniquesÐthree purely static techniques and two hybrid static-dynamic techniquesÐthat aim to make static RTS safe with respect to reflection. We implement these reflection-aware (RA) techniques by extending the reflection-unaware (RU) class-level static RTS technique in a tool called STARTS. To evaluate these RA techniques, we compare their end-to-end times with RU, and with RetestAll, which reruns all tests after every code change. We also compare safety and precision of the RA techniques with Ekstazi, a state-of-the-art dynamic RTS technique; precision is a measure of unaffected tests selected. Our evaluation on 1173 versions of 24 open-source Java projects shows negative results. The RA techniques improve the safety of RU but at very high costs. The purely static techniques are safe in our experiments but decrease the precision of RU, with end-to-end time at best 85.8% of RetestAll time, versus 69.1% for RU. One hybrid static-dynamic technique improves the safety of RU but at high cost, with end-to-end time that is 91.2% of RetestAll. The other hybrid static-dynamic technique provides better precision, is safer than RU, and incurs lower end-to-end timeÐ75.8% of RetestAll, but it can still be unsafe in the presence of test-order dependencies. Our study highlights the challenges involved in making static RTS safe with respect to reflection. CCS Concepts: • Software and its engineering → Automated static analysis; Software testing and debugging; Software evolution.

show abstract

“…All paths begin with a require(n) action, where n is the name of a Node.js module. 7 The require(n) action can be followed by a sequence of property reads (denoted .n where n is a property name), function and constructor applications (denoted () κ and new () κ where κ is explained below) and argument reads (denoted κ →arg j where j indicates the zero-indexed position of the argument). We refer to Mezzetti et al [15] for further description of these different kinds of actions that also appear in NoRegrets.…”

Section: Phase I: Model Generationmentioning

confidence: 99%

“…The study by Gyori et al [7] used client test suites to detect breaking changes in library updates, similar to the dont-break methodology mentioned in Section 1 but for Java. They note that it is common practice in industry to use this form of testing, but also that applying certain test case selection criteria could yield a considerable speed-up while preserving coverage, similar to how NoRegrets+ avoids running all the client test suites at every library update.…”

Section: Studies Of Breaking Changes In Library Updatesmentioning

confidence: 99%

Model-based testing of breaking changes in Node.js libraries

Møller

Torp

2019

Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of

View full text Add to dashboard Cite

Semantic versioning is widely used by library developers to indicate whether updates contain changes that may break existing clients. Especially for dynamic languages like JavaScript, using semantic versioning correctly is known to be difficult, which often causes program failures and makes client developers reluctant to switch to new library versions. The concept of type regression testing has recently been introduced as an automated mechanism to assist the JavaScript library developers. That mechanism is effective for detecting breaking changes in widely used libraries, but it suffers from scalability limitations that make it slow and also less useful for libraries that do not have many available clients. This paper presents a model-based variant of type regression testing. Instead of comparing API models of a library before and after an update, it finds breaking changes by automatically generating tests from a reusable API model. Experiments show that this new approach significantly improves scalability: it runs faster, and it can find breaking changes in more libraries. CCS CONCEPTS • Software and its engineering → Software maintenance tools; Software testing and debugging.

show abstract

Evaluating Regression Test Selection Opportunities in a Very Large Open-Source Ecosystem

Cited by 23 publications

References 33 publications

Bridging the model-to-code abstraction gap with fuzzy logic in model-based regression test selection

Bridging the model-to-code abstraction gap with fuzzy logic in model-based regression test selection

Reflection-aware static regression test selection

Model-based testing of breaking changes in Node.js libraries

Contact Info

Product

Resources

About