Detecting Assumptions on Deterministic Implementations of Non-deterministic Specifications

NonDex: a tool for detecting and debugging wrong assumptions on Java API specifications

“…The NonDex Maven plugin provides three goals: nondex:nondex runs all tests while exploring different random behaviors of 41 underdetermined APIs from the standard Java library and reports failing tests; nondex:debug reruns test(s) that failed during exploration to locate the invocation(s) of an underdetermined API that made the wrong assumption; and nondex:clean deletes the .nondex directories where NonDex stores auxiliary files during exploration and debugging. The NonDex Maven plugin also takes several optional arguments (with sensible defaults): nondexSeed is the main seed to use for randomization; nondexRuns is the number of runs to perform, each with a different starting seed computed from the main seed; and nondexMode is one of two exploration modes, ONE and FULL, where ONE has more determinism than FULL as described in our prior work [16].…”

“…The challenge is to develop instrumentation that can automatically handle a large number of Java versions. For our original prototype [16], we manually modified the Java sources of the relevant files, compiled them, and used them in place of the original files. However, this solution was brittle, because the tool would often not work unless the exact same Java version (e.g., 1.8.0-b132) was used for the run as the version for which we manually modified the sources.…”

“…NonDex currently supports two kinds of behaviors: (1) permutation for APIs where order is underdetermined, and (2) extensions for APIs where only lower bounds on array size(s) are specified. The runner takes as inputs (i) a random seed, which completely determines the choice of behaviors, (ii) the mode of exploration-ONE or FULL (the two modes differ in the kind of wrong assumptions they can detect, as described in detail in our prior work [16] 3 ), and (iii) optionally the range of choices to be randomized (which is used by debugging).…”

“…The debugging phase reports to the user an API that causes the detected failure together with the call stack of the API's invocation which further helps in localizing the context in which the wrong assumption was made. In our prior work [16], we performed all debugging manually; after implementing automated debugging, we found that we had made an error in manually identifying the root cause of one failure, which shows that the automated debugging helps to more reliably identify the root causes.…”

“…This paper makes several contributions to the tool design and implementation. We recently published the research idea of NonDex [16] and evaluated it using a prototype implementation which modified the OpenJDK JVM and was not easily portable. The current NonDex tool improves four aspects: (1) the tool is pure Java and is widely portable due to a novel instrumentation mechanism, (2) the Maven plugin is new and allowed us to scale our experiments, (3) the debugging feature is also new, and (4) NonDex is now open source on GitHub and is under active development [15].…”