The Rode0day to Less-Buggy Programs

Fasano, Andrew; Leek, Tim; Dolan-Gavitt, Brendan; Bundt, Josh

doi:10.1109/msec.2019.2933682

Cited by 6 publications

(5 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Of the three synthetic bug-injection systems mentioned above, LAVA was the only one to release publicly available corpora of buggy programs: LAVA-M and LAVA-1. The LAVA-M corpus is commonly used for fuzzing evaluation in the literature [12]. LAVA builds off of the PANDA [10] whole-system dynamic analysis platform to conduct a dynamic taint analysis on input files ingested by target applications.…”

Section: Lava-based Bug Injectionmentioning

confidence: 99%

See 1 more Smart Citation

Evaluating Synthetic Bugs

Bundt

Fasano

Dolan-Gavitt

et al. 2021

Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security

Self Cite

View full text Add to dashboard Cite

Fuzz testing has been used to find bugs in programs since the 1990s, but despite decades of dedicated research, there is still no consensus on which fuzzing techniques work best. One reason for this is the paucity of ground truth: bugs in real programs with known root causes and triggering inputs are difficult to collect at a meaningful scale. Bug injection technologies that add synthetic bugs into real programs seem to offer a solution, but the differences in finding these synthetic bugs versus organic bugs have not previously been explored at a large scale. Using over 80 years of CPU time, we ran eight fuzzers across 20 targets from the Rode0day bug-finding competition and the LAVA-M corpus. Experiments were standardized with respect to compute resources and metrics gathered. These experiments show differences in fuzzer performance as well as the impact of various configuration options. For instance, it is clear that integrating symbolic execution with mutational fuzzing is very effective and that using dictionaries improves performance. Other conclusions are less clear-cut; for example, no one fuzzer beat all others on all tests. It is noteworthy that no fuzzer found any organic bugs (i.e., one reported in a CVE), despite 50 such bugs being available for discovery in the fuzzing corpus. A close analysis of results revealed a possible explanation: a dramatic difference between where synthetic and organic bugs live with respect to the "main path" discovered by fuzzers. We find that recent updates to bug injection systems have made synthetic bugs more difficult to discover, but they are still significantly easier to find than organic bugs in our target programs. Finally, this study identifies flaws in bug injection techniques and suggests a number of axes along which synthetic bugs should be improved. DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited. This material is based upon work supported by the Department of Defense under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Department of Defense.

show abstract

Section: Lava-based Bug Injectionmentioning

confidence: 99%

“…Luckily, the Rode0day corpus satisfies each of these criteria. Rode0day [12] is a continuous bug-finding competition that uses synthetic bugs. The Rode0day corpus 3 is a collection of 55 challenges built from 12 programs deployed in previous competitions.…”

Section: Challenge Corpusmentioning

confidence: 99%

Evaluating Synthetic Bugs

Bundt

Fasano

Dolan-Gavitt

et al. 2021

Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, OSS-FUZZ [15,16] or recent greybox fuzzers [25,62,73] only found a small number of UAF. Actually, RODE0DAY [17], a continuous bug finding competition, recognizes that fuzzers should aim to cover new bug classes like UAF in the future [37], moving further from the widely-used LAVA [36] bug corpora which only contains buffer overflows.…”

Section: Introductionmentioning

confidence: 99%

Binary-level Directed Fuzzing for Use-After-Free Vulnerabilities

Nguyen¹,

Bardin²,

Bonichon³

et al. 2020

Preprint

View full text Add to dashboard Cite

Directed fuzzing focuses on automatically testing specific parts of the code by taking advantage of additional information such as (partial) bug stack trace, patches or risky operations. Key applications include bug reproduction, patch testing and static analysis report verification. Although directed fuzzing has received a lot of attention recently, hard-to-detect vulnerabilities such as Use-After-Free (UAF) are still not well addressed, more especially at the binary level. We propose UAFUZZ, the first (binary-level) directed greybox fuzzer dedicated to UAF bugs. The technique features a fuzzing engine tailored to UAF specifics, a lightweight code instrumentation and an efficient bug triage step. Experimental evaluation for bug reproduction on real cases demonstrates that UAFUZZ significantly outperforms state-of-the-art directed fuzzers in terms of fault detection rate, time to exposure and bug triaging. UAFUZZ has also been proven effective in patch testing, leading to the discovery of 20 new bugs in Perl, GPAC and GNU Patch (including a buggy patch) -all of them have been acknowledged and 14 have been fixed. Last but not least, we provide to the community the first fuzzing benchmark dedicated to UAF, built on both real codes and real bugs.

show abstract

“…Luckily, the Rode0day corpus satisfies each of these criteria. Rode0day [100] is a continuous bug-finding competition that uses synthetic bugs. The Rode0day corpus 4 is a collection of 55 challenges built from 12 programs deployed in previous competitions.…”

Section: Test Corpora and Methodologymentioning

confidence: 99%

“…Of the three synthetic bug-injection systems mentioned above, LAVA was the only one to release publicly available corpora of buggy programs: LAVA-M and LAVA-1. The LAVA-M corpus is commonly used for fuzzing evaluation in the literature [100]. LAVA builds off of the PANDA [41] whole-system dynamic analysis platform to conduct a dynamic taint analysis on input files ingested by target applications.…”

Section: Lava-based Bug Injectionmentioning

confidence: 99%

Towards rigorous evaluation of binary testing and analysis

Bundt

View full text Add to dashboard Cite

Towards Rigorous Evaluation of Binary Testing and Analysis by Joshua BundtComputer security research is an ever-evolving field that aims to make technology more secure.Attackers constantly seek out vulnerabilities in systems, and defenders strive to introduce new controls to prevent these attacks. Attack research typically involves demonstrating the validity of an attack through a proof of concept. In contrast, defense research requires a higher level of rigor to substantiate that defenses are secure under various conditions and against a willful adversary. In this thesis, we examine the state of rigor in a specific area of defense research: binary testing and analysis. Binary testing and analysis encompasses the tasks and techniques required to evaluate binary code, which is the machine-readable representation of software programs, in order to understand program behavior, identify vulnerabilities, and ensure correctness and security. To assess the robustness of the current techniques and to provide a more rigorous methodology, we first examine the utility of synthetic bug generation as a solution to the scarcity of real bugs for fuzz testing evaluation. We conducted a large-scale measurement study evaluating existing synthetic bug generators with eight fuzzers on 20 software libraries and found that synthetic bugs are easier to discover than organic bugs and the most popular synthetic bug benchmark, LAVA-M, exhibits fundamental flaws that make it unsuitable to recommend for future research. Second, we propose a new workflow to enable humans to more effectively assist fuzz testing through compartment analysis. An empirical study of seven software libraries revealed that compartment analysis can significantly improve a fuzzing campaign even when conducted after a few hours of fuzzing. Finally, we consider the fragility of neural network binary disassemblers at the task of function boundary detection. When comparing traditional disassemblers to neural binary disassemblers, we found the latter to be vulnerable to adversarial attacks which allows the attacker to degrade function boundary detection. In response, we proposed an expanded set of benchmarks and adversarial techniques to provide a better evaluation of neural binary disassemblers. Throughout this dissertation, we propose and demonstrate improved methodologies for rigorously examining and assessing binary testing and analysis efficacy. v AcknowledgementsThe PhD journey starts with someone agreeing to take a long term risk on you despite knowing little more than your resume. For taking the initial risk, I would like to thank my advisor Wil Robertson who has guided me from start to finish while providing advice, unwaverable optimism, and tremendous patience. A special thanks to Tim Leek who took on mentoring me long before I started at Northeastern and continues to provide keen insight and frank opinions that shape my research efforts. I would also like to thank the rest of the members of my committee Pete Manolios, Guevara Noubir, and Davide Balzarotti who pr...

show abstract

The Rode0day to Less-Buggy Programs

Cited by 6 publications

References 3 publications

Evaluating Synthetic Bugs

Evaluating Synthetic Bugs

Binary-level Directed Fuzzing for Use-After-Free Vulnerabilities

Towards rigorous evaluation of binary testing and analysis

Contact Info

Product

Resources

About