QuickFuzz testing for fun and profit

Grieco, Gustavo; Ceresa, Martín; Mista, Agustín; Buiras, Pablo

doi:10.1016/j.jss.2017.09.018

Cited by 17 publications

(13 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Skyfire [Wang et al 2017] and Orthrus [Shastry et al 2017] do this by generating well-formed initial seeds, according to a probabilistic context-sensitive grammar inferred from real-world examples. QuickFuzz [Grieco et al 2016[Grieco et al , 2017 allows seed generation through the use of grammars that specify the structure of valid, or interesting, inputs (mainly file formats). DIFUZE [Corina et al 2017] performs an up-front static analysis to identify the structure of inputs to device drivers prior to fuzzing.…”

Section: Coverage-guided Fuzzingmentioning

confidence: 99%

Coverage guided, property based testing

Lampropoulos

Hicks

Pierce

2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

Property-based random testing, exemplified by frameworks such as Haskell's QuickCheck, works by testing an executable predicate (a property) on a stream of randomly generated inputs. Property testing works very well in many cases, but not always. Some properties are conditioned on the input satisfying demanding semantic invariants that are not consequences of its syntactic structureÐe.g., that an input list must be sorted or have no duplicates. Most randomly generated inputs fail to satisfy properties with such sparse preconditions, and so are simply discarded. As a result, much of the target system may go untested. We address this issue with a novel technique called coverage guided, property based testing (CGPT). Our approach is inspired by the related area of coverage guided fuzzing, exemplified by tools like AFL. Rather than just generating a fresh random input at each iteration, CGPT can also produce new inputs by mutating previous ones using type-aware, generic mutation operators. The target program is instrumented to track which control flow branches are executed during a run and inputs whose runs expand control-flow coverage are retained for future mutations. This means that, when sparse conditions in the target are satisfied and new coverage is observed, the input that triggered them will be retained and used as a springboard to go further. We have implemented CGPT as an extension to the QuickChick property testing tool for Coq programs; we call our implementation FuzzChick. We evaluate FuzzChick on two Coq developments for abstract machines that aim to enforce flavors of noninterference, which has a (very) sparse precondition. We systematically inject bugs in the machines' checking rules and use FuzzChick to look for counterexamples to the claim that they satisfy a standard noninterference property. We find that vanilla QuickChick almost always fails to find any bugs after a long period of time, as does an earlier proposal for combining property testing and fuzzing. In contrast, FuzzChick often finds them within seconds to minutes. Moreover, FuzzChick is almost fully automatic; although highly tuned, handwritten generators can find the bugs faster than FuzzChick, they require substantial amounts of insight and manual effort. CCS Concepts: • Software and its engineering → General programming languages.

show abstract

Section: Coverage-guided Fuzzingmentioning

confidence: 99%

Coverage guided, property based testing

Lampropoulos

Hicks

Pierce

2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

show abstract

“…The second approach we will discuss is the one taken by MegaDeTH , a meta-programming tool used intensively by QuickFuzz [14,15]. Firstly, MegaDeTH derives random generators for ADTs as well as all of its nested types-a useful feature not supported by derive.…”

Section: Megadethmentioning

confidence: 99%

“…For our experiments, we use the coverage measure known as execution path employed by American Fuzzy Lop (AFL) [21]-a well known fuzzer. It was chosen in this work since it is also used in the work by Grieco et al [15] to compare MegaDeTH with other techniques. The process consists of the instrumentation of the binaries under test, making them able to return the path in the code taken by each execution.…”

Section: Case Studiesmentioning

confidence: 99%

“…Let C u i and C v j be type constructors of T u and T v respectively. Then, by (13) and (15) we have:…”

Section: A Demonstrationsmentioning

confidence: 99%

“…As a result, generated random values are well-typed and preserve the structure described by the ADT. Rather than manually writing generators, libraries derive [24] and MegaDeTH [14,15] automatically synthesize generators for a given user-defined ADT. The library derive provides no guarantees that the generation process terminates, while MegaDeTH pays almost no attention to the distribution of values.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Branching processes for QuickCheck generators

Mista

Russo

Hughes

2018

Proceedings of the 11th ACM SIGPLAN International Symposium on Haskell

Self Cite

View full text Add to dashboard Cite

In QuickCheck (or, more generally, random testing), it is challenging to control random data generators' distributionsspecially when it comes to user-defined algebraic data types (ADT). In this paper, we adapt results from an area of mathematics known as branching processes, and show how they help to analytically predict (at compile-time) the expected number of generated constructors, even in the presence of mutually recursive or composite ADTs. Using our probabilistic formulas, we design heuristics capable of automatically adjusting probabilities in order to synthesize generators which distributions are aligned with users' demands. We provide a Haskell implementation of our mechanism in a tool called and perform case studies with real-world applications. When generating random values, our synthesized QuickCheck generators show improvements in code coverage when compared with those automatically derived by state-of-the-art tools.CCS Concepts • Software and its engineering → Software testing and debugging;

show abstract