Testing an optimising compiler by generating random lambda terms

Abstract

This paper considers random testing of a compiler, using randomly generated programs as inputs, and comparing their behaviour with and without optimisation. Since the generated programs must compile, then we need to take into account syntax, scope rules, and type checking during our random generation. Doing so, while attaining a good distribution of test data, proves surprisingly subtle; the main contribution of this paper is a workable solution to this problem. We used it to generate typed functions on lists, which we compiled using the Glasgow Haskell compiler, a mature production quality Haskell compiler. After around 20,000 tests we triggered an optimiser failure, and automatically simplified it to a program with just a few constructs.

Beginner's luck: a language for property-based generators

“…This is a slowdown of 20x compared to that of Pałka's. However, our implementation is a total of 82 lines of fairly simple code, while the handwritten development is 1684 lines, with the warning "...the code is difficult to understand, so reading it is not recommended" in its distribution page [59].…”

“…For example, papers have been written about random generation techniques for well-typed lambda-terms [25,60,64,68] and for "indistinguishable" machine states that can be used for finding bugs in information-flow monitors [39,40]. Moreover, if we use QuickCheck to test an invariant property (e.g., type preservation), then the same condition will appear in both the precondition and the conclusion of the property, requiring that we express this condition both as a boolean predicate p and as a generator whose outputs all satisfy p. These two artifacts must then be kept in sync, which can become both a maintenance issue and a rich source of confusion in the testing process.…”

“…Moreover, if we use QuickCheck to test an invariant property (e.g., type preservation), then the same condition will appear in both the precondition and the conclusion of the property, requiring that we express this condition both as a boolean predicate p and as a generator whose outputs all satisfy p. These two artifacts must then be kept in sync, which can become both a maintenance issue and a rich source of confusion in the testing process. These difficulties are not hypothetical: Hriţcu et al's machine-state generator [39] is over 1500 lines of tricky Haskell, while Pałka et al's generator for welltyped lambda-terms [60] is over 1600 even trickier ones. To enable effective property-based random testing of complex software artifacts, we need a better way of writing predicates and corresponding generators.…”

Feat

“…It would be worthwhile to try and falsify or confirm the general understanding for instance by attempting to repeat the results of [Pałka et al 2011] using systematic enumeration.…”

“…Similar to SmallCheck, the enumeration in [Yakushev and Jeuring 2009] adopts the list view, which prohibits the sampling of large values. On the other hand, the special-purpose QuickCheck generator designed in [Pałka et al 2011], randomly generates welltyped terms. Unsurprisingly, it has no problem with constructing individual large terms, but falls short in systematicness.…”