Improving oracle quality by detecting brittle assertions and unused inputs in tests

Abstract

Writing oracles is challenging. As a result, developers often create oracles that check too little, resulting in tests that are unable to detect failures, or check too much, resulting in tests that are brittle and difficult to maintain. In this paper we present a new technique for automatically analyzing test oracles. The technique is based on dynamic tainting and detects both brittle assertions-assertions that depend on values that are derived from uncontrolled inputs-and unused inputs-inputs provided by the test that are not checked by an assertion. We also presented OraclePolish, an implementation of the technique that can analyze tests that are written in Java and use the JUnit testing framework. Using OraclePolish, we conducted an empirical evaluation of more than 4000 real test cases. The results of the evaluation show that OraclePolish is effective; it detected 164 tests that contain brittle assertions and 1618 tests that have unused inputs. In addition, the results also demonstrate that the costs associated with using the technique are reasonable.

Reliable testing: detecting state-polluting tests to prevent test dependency

“…PT sets the field testDir (line 22) and leaves it polluted. In that revision, no other test read the value of testDir, so no existing technique (e.g., Zhang et al's technique [28] based on test reordering or Huo and Clause's technique [7] based on taint analysis) would report PT as a polluting test.…”

“…Huo and Clause [7] use taint analysis to find brittle assertions, i.e., cases when a test reads from state regions not explicitly written by the test. These reads can find potential test dependencies on heap-shared state.…”

“…One common problem [1,7,13,18,23,28] in regression test suites is dependency between tests. These dependencies arise when the tests read and write some shared resource, e.g., the heap state in the main memory, file system, database, etc.…”

Dynamic taint tracking for Java with phosphor (demo)

“…By operating within the JVM, taint tracking systems can leverage language semantics that greatly simplify memory organization (such as variables). However, in Java, associating metadata (such as tags) with arbitrary variables is very difficult: previous techniques have relied on customized JVMs or symbolic execution environments to maintain this mapping [3,11,13], limiting their portability and restricting their application to large and complex real-world software.…”

“…Inputs to the application being studied are labeled with a marker (are "tainted"), and these markers propagated through data flow. Dynamic taint tracking can be used for detecting brittle tests [11], end user privacy testing [7,14] and debugging [8,12].…”