Unit test virtualization with VMVM

Abstract

Testing large software packages can become very time intensive. To address this problem, researchers have investigated techniques such as Test Suite Minimization. Test Suite Minimization reduces the number of tests in a suite by removing tests that appear redundant, but risks a reduction in fault-finding ability since it can be difficult to identify which tests are truly redundant. We take a completely different approach to solving the same problem of long running test suites by instead reducing the time needed to execute each test, an approach that we call Unit Test Virtualization. We describe the empirical analysis that we performed to ground our approach and provide an implementation of Unit Test Virtualization targeting Java applications. We evaluated our implementation, VmVm, using 20 real-world Java applications and found that it reduces test suite execution time by up to 97% (on average, 62%) when compared to traditional unit test execution. We also compared VmVm to a well known Test Suite Minimization technique, finding the reduction provided by VmVm to be four times greater, while still executing every test with no loss of fault-finding ability.

Notice of Retraction: Does Refactoring of Test Smells Induce Fixing Flaky Tests?

“…As pointed out by several researchers and practitioners, flaky tests represent an important issue for regression testing [18], [25], [14], [20], [21], [22]. Memon and Cohen [20] described a set of negative effects that flaky tests may create during regression testing, finding that their presence can even lead to missing deadlines due to the fact that certain features cannot be tested sufficiently [20].…”

“…For instance, Zhang et al [22] focused on test suites affected by test dependency issues, by reporting methodologies to identify these tests. At the same time, Muslu et al [21] found that test isolation may help in fault localization, while Bell and Kaiser [25] proposed a technique able to isolate test cases in Java applications by tracking the shared objects in memory.…”

“…Perhaps most importantly, from a psychological point of view flaky tests can reduce a developer's confidence in the tests, possibly leading to ignoring actual test failures [17]. Because of this, the research community has spent considerably effort on trying to understand the causes behind test flakiness [18], [20], [21], [22] and on devising automated techniques able to fix flaky tests [23], [24], [25]. However, most of this research mainly focused the attention on some specific causes possibly leading to the introduction of flaky tests, such as concurrency [26], [25], [27] or test order dependency [22] issues, thus proposing ad-hoc solutions that cannot be used to fix flaky tests characterized by other root causes.…”

“…Because of this, the research community has spent considerably effort on trying to understand the causes behind test flakiness [18], [20], [21], [22] and on devising automated techniques able to fix flaky tests [23], [24], [25]. However, most of this research mainly focused the attention on some specific causes possibly leading to the introduction of flaky tests, such as concurrency [26], [25], [27] or test order dependency [22] issues, thus proposing ad-hoc solutions that cannot be used to fix flaky tests characterized by other root causes. Indeed, according to the findings by Luo et al [18] who conducted an empirical study on the motivations behind test code flakiness, the problems faced by previous research only represent a part of whole story: a deeper analysis of possible fixing strategies of other root causes (e.g., flakiness due to wrong usage of external resources) is still missing.…”

Fine-Grained Test Minimization

“…A number of techniques address the problem of long running test suites by running tests in virtualized [4], or cloud-based parallel [14,25] environments. Any of these techniques require complete test independency, a requirement hard to meet in many real-world test suites including those of our subject systems, which we ran sequentially.…”