Inputs From Hell:

Abstract

Grammars can serve as producers for structured test inputs that are syntactically correct by construction. A probabilistic grammar assigns probabilities to individual productions, thus controlling the distribution of input elements. Using the grammars as input parsers, we show how to learn input distributions from input samples, allowing to create inputs that are similar to the sample; by inverting the probabilities, we can create inputs that are dissimilar to the sample. This allows for three test generation strategies: 1) "Common inputs" -by learning from common inputs, we can create inputs that are similar to the sample; this is useful for regression testing. 2) "Uncommon inputs" -learning from common inputs and inverting probabilities yields inputs that are strongly dissimilar to the sample; this is useful for completing a test suite with "inputs from hell" that test uncommon features, yet are syntactically valid. 3) "Failure-inducing inputs" -learning from inputs that caused failures in the past gives us inputs that share similar features and thus also have a high chance of triggering bugs; this is useful for testing the completeness of fixes. Our evaluation on three common input formats (JSON, JavaScript, CSS) shows the effectiveness of these approaches. Results show that "common inputs" reproduced 96% of the methods induced by the samples. In contrast, for almost all subjects (95%), the "uncommon inputs" covered significantly different methods from the samples. Learning from failure-inducing samples reproduced all exceptions (100%) triggered by the failure-inducing samples and discovered new exceptions not found in any of the samples learned from.

FormatFuzzer: Effective Fuzzing of Binary File Formats

“…Black-box strategies. Recent advances in grammar-based fuzzing, such as systematically achieving grammar coverage [22] or learning and leveraging probability distributions [39] could easily be adopted for binary template fuzzing, too. Search-based fuzzing.…”

Grammar-based Fuzzing of Data Integration Parsers in Computational Materials Science

“…This technique is aimed at provoking erroneous program behavior. However, when working on a program that uses highly structured input data, just randomly generating inputs will result in only testing the input validation of the program under test (PUT) 13 . To overcome this limitation, grammar based fuzzers (e.g., Nautilus 14 , EvoGFuzz 15 , syntax machine 16 ) were developed, that make use of a grammar for the structured inputs.…”

“…Furthermore, researchers employed probabilistic grammars in order to target specific regions of the PUT. These probabilistic grammars may be used to generate files that execute specific branches while being evaluated 13 . Using this technique, the fuzzer is able to generate test inputs that are (by definition of the grammar) syntactically valid, but can also be aimed at specific areas of the PUT that are under-tested according to some metric.…”

Grammar‐based fuzzing of data integration parsers in computational materials science

“…These probabilistic grammars may be used to generate files that execute specific branches while being evaluated. 13 Using this technique, the fuzzer is able to generate test inputs that are (by definition of the grammar) syntactically valid, but can also target specific areas of the PUT that are under-tested according to some metric. Therefore, (probabilistic) grammar-based fuzzing is a sub-form of white-box fuzzing, in which a model of the PUT is available to the fuzzer.…”