QuickFuzz: an automatic random fuzzer for common file formats

Abstract

Fuzzing is a technique that involves testing programs using invalid or erroneous inputs. Most fuzzers require a set of valid inputs as a starting point, in which mutations are then introduced. QuickFuzz is a fuzzer that leverages QuickCheck-style random test-case generationto automatically test programs that manipulate common file formats by fuzzing. We rely on existing Haskell implementations of file-format-handling libraries found on Hackage, the community-driven Haskell code repository. We have tried QuickFuzz in the wild and found that the approach is effective in discovering vulnerabilities in real-world implementations of browsers, image processing utilities and file compressors among others. In addition, we introduce a mechanism to automatically derive random generators for the types representing these formats. QuickFuzz handles most well-known image and media formats, and can be used to test programs and libraries written in any language.

Evaluating Fuzz Testing

“…Skyfire [53] and Orthrus [49] propose to improve the initial seed selection by running an up-front analysis on the program to bootstrap information both for creating the corpus and assisting the mutators. QuickFuzz [20,21] allows seed generation through the use of grammars that specify the structure of valid, or interesting, inputs. DIFUZE performs an up-front static analysis to identify the structure of inputs to device drivers prior to fuzzing [13].…”

“…In Table 1, we can see that 12 out of the 15 papers that did consider multiple trials did not characterize the performance variance (they have a blank box in the variance column). Instead, each of them compared the "average" performance (we assume: arithmetic mean) of A and B when drawing conclusions, except for Dowser [22] that reported median, and two [20,59] that did not mention how the "average" was calculated.…”

Branching processes for QuickCheck generators

“…The second approach we will discuss is the one taken by MegaDeTH , a meta-programming tool used intensively by QuickFuzz [14,15]. Firstly, MegaDeTH derives random generators for ADTs as well as all of its nested types-a useful feature not supported by derive.…”

“…Fuzzers are tools to tests programs against randomly generated unexpected inputs. QuickFuzz [14,15] is a tool that synthesizes data with rich structure, that is, well-typed files which can be used as initial "seeds" for state-of-the-art fuzzers-a work flow which discovered many unknown vulnerabilities. Our work could help to improve the variation of the generated initial seeds, by varying the distribution of QuickFuzz generators-an interesting direction for future work.…”

“…As a result, generated random values are well-typed and preserve the structure described by the ADT. Rather than manually writing generators, libraries derive [24] and MegaDeTH [14,15] automatically synthesize generators for a given user-defined ADT. The library derive provides no guarantees that the generation process terminates, while MegaDeTH pays almost no attention to the distribution of values.…”