A smart fuzzing method for detecting heap‐based vulnerabilities in executable codes

Abstract

In this paper, we present a smart fuzzing method for detecting six classes of heap‐based vulnerabilities in executable codes, that is, heap‐based buffer overflow, buffer underwrite, buffer over‐read, buffer under‐read, double‐free, and use‐after‐free vulnerabilities. In the proposed method, the executable code is instrumented to perform concolic (concrete + symbolic) execution and calculate the constraints on input data for executing a specific path. We also define a number of vulnerability constraints that determine the characteristics of input data that activate specific vulnerabilities in an execution path. By calculating symbolic path and vulnerability constraints in each executed path, we generate appropriate concrete input data that traverse other paths in the program and detect the vulnerabilities in the executed paths. We have implemented the proposed smart fuzzing method as a plug‐in for Valgrind framework and tested it on different groups of benchmark programs. The results demonstrate that the calculated vulnerability constraints are accurate, and our fuzzer is able to detect the vulnerabilities in these programs. We have also compared the implemented fuzzer with three other fuzzers and demonstrated how calculating the path and vulnerability constraints in our method helps to fuzz a program more efficiently. Copyright © 2016 John Wiley & Sons, Ltd.