VMVM: unit test virtualization for Java

Abstract

As software evolves and grows, its regression test suites tend to grow as well. When these test suites become too large, they can eventually reach a point where they become too length to regularly execute. Previous work in Test Suite Minimization has reduced the number of tests in such suites by attempting to identify those that are redundant (e.g. by a coverage metric). Our approach to ameliorating the runtime of these large test suites is complementary, instead focusing on reducing the overhead of running each test, an approach that we call Unit Test Virtualization. This Tool Demonstration presents our implementation of Unit Test Virtualization, VmVm (pronounced "vroom-vroom") and summarizes an evaluation of our implementation on 20 real-world Java applications, showing that it reduces test suite execution time by up to 97% (on average, 62%). A companion video to this demonstration is available online, at https://www.youtube.com/watch?v=sRpqF3rJERI.

Unit test virtualization with VMVM

“…We are interested in exploring further the technical challenges of implementing Unit Test Virtualization in non-memory managed languages, as well as the potential for additional performance gains by combining Unit Test Virtualization directly with Test Suite Minimization. VmVm is currently available publicly on GitHub [7], and could potentially be applied to other domains such as security or fault-recovery by providing a lightweight automatic reinitialization feature. Table 7: Complete listing of applications studied, continues to next page N N Table 7: Complete listing of applications studied, continues to next page…”

“…We call our implementation VmVm, pronounced "vroomvroom," and name it after its technique of building a Virtual Machine-like container within the Java Virtual Machine). VmVm is released under an MIT license and is available on GitHub [7]. We integrated VmVm directly with the popular test utility JUnit and two common build systems: ant and maven, to reset the test environment between automated test executions with no intervention.…”

“…efficiently isolate test cases that is language agnostic, certainly among memory managed languages 3. An implementation of our technique for Java, VmVm (released freely via GitHub [7]), evaluated to show its efficacy in reducing test suite runtime and maintaining fault-finding properties Section 2 explains the factors that contribute to the need for Unit Test Virtualization as well as our study showing the applicability of our approach to large Java applications. Section 3 describes the general Unit Test Virtualization approach.…”

“…We implemented Unit Test Virtualization for Java, creating our tool VmVm (pronounced "vroom-vroom"), which transforms application byte code directly without requiring modification to the JVM or access to application source code. We have integrated it directly with popular Java testing and build automation tools JUnit [2], ant [5] and maven [6], and it is available for download via GitHub [7].…”

Detecting, isolating, and enforcing dependencies among and within test cases

“…In addition to our relevant prior work that will be discussed elsewhere in this proposal (Phosphor [1], VMVM [2,3] and Chronicler [4]), we also have several other publications on the unrelated topic of gamification, included here for completeness: [5,6,15].…”