UNIFUZZ: A Holistic and Pragmatic Metrics-Driven Platform for Evaluating Fuzzers

Abstract

A flurry of fuzzing tools (fuzzers) have been proposed in the literature, aiming at detecting software vulnerabilities effectively and efficiently. To date, it is however still challenging to compare fuzzers due to the inconsistency of the benchmarks, performance metrics, and/or environments for evaluation, which buries the useful insights and thus impedes the discovery of promising fuzzing primitives. In this paper, we design and develop UNIFUZZ, an open-source and metrics-driven platform for assessing fuzzers in a comprehensive and quantitative manner. Specifically, UNIFUZZ to date has incorporated 35 usable fuzzers, a benchmark of 20 real-world programs, and six categories of performance metrics. We first systematically study the usability of existing fuzzers, find and fix a number of flaws, and integrate them into UNIFUZZ. Based on the study, we propose a collection of pragmatic performance metrics to evaluate fuzzers from six complementary perspectives. Using UNIFUZZ, we conduct in-depth evaluations of several prominent fuzzers including AFL VUzzer64 [8]. We find that none of them outperforms the others across all the target programs, and that using a single metric to assess the performance of a fuzzer may lead to unilateral conclusions, which demonstrates the significance of comprehensive metrics. Moreover, we identify and investigate previously overlooked factors that may significantly affect a fuzzer's performance, including instrumentation methods and crash analysis tools. Our empirical results show that they are critical to the evaluation of a fuzzer. We hope that our findings can shed light on reliable fuzzing evaluation, so that we can discover promising fuzzing primitives to effectively facilitate fuzzer designs in the future.Yuwei Li and Shouling Ji are the co-first authors. Shouling Ji and Chunming Wu are the co-corresponding authors.

Towards rigorous evaluation of binary testing and analysis

“…This period of stunted progress had a deluge of academic papers which claimed superiority over AFL [46], [65], [69], [72], [83], [110], [121], with the last algorithmic update in 2017 [201]. Despite the appearance of progress, several recent papers have independently evaluated AFL with many of its derivatives and concluded that "superiority" is marginal [170], [182]. Many of the claimed victories were limited to specific conditions of the evaluation methodology, but failed to generalize when other researchers completed independent testing.…”

“…In the discipline of machine learning, Dehghani et al [166] examines the critical importance of canonical benchmarks and how the bias of benchmarks shapes the collective decisions about which algorithms are better while perhaps losing perspective on the implications of the assumed bias in the benchmarks. Other studies examine the biases within existing benchmarks or propose new benchmarks [141], [170], [183].…”

“…Additionally, some conferences, including top-tier security conferences have added optional artifact evaluation to address the issues of reproducibility, verifiability, and transparency. Finally, some papers primarily attempt to reproduce previous research in order to provide comprehensive evaluations that compares many existing systems using common benchmarks, metrics, and experimental methodology [170], [191]. To that end, the field must establish standards, identify potential pitfalls, and critically examine prior work to help researchers more precisely assess the effectiveness of research efforts and advance the maturity of the field.…”