Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source

Abstract

Deep learning (DL) systems can make our life much easier, and thus is gaining more and more attention from both academia and industry. Meanwhile, bugs in DL systems can be disastrous, and can even threaten human lives in safety-critical applications. To date, a huge body of research efforts have been dedicated to testing DL models. However, interestingly, there is still limited work for testing the underlying DL libraries, which are the foundation for building, optimizing, and running the DL models. One potential reason is that test generation for the underlying DL libraries can be rather challenging since their public APIs are mainly exposed in Python, making it even hard to automatically determine the API input parameter types due to dynamic typing. In this paper, we propose FreeFuzz, the first approach to fuzzing DL libraries via mining from open source. More specifically, FreeFuzz obtains code/models from three different sources: 1) code snippets from the library documentation, 2) library developer tests, and 3) DL models in the wild. Then, FreeFuzz automatically runs all the collected code/models with instrumentation to trace the dynamic information for each covered API, including the types and values of each parameter during invocation, and shapes of input/output tensors. Lastly, FreeFuzz will leverage the traced dynamic information to perform fuzz testing for each covered API. The extensive study of FreeFuzz on PyTorch and TensorFlow, two of the most popular DL libraries, shows that FreeFuzz is able to automatically trace valid dynamic information for fuzzing 1158 popular APIs, 9X more than state-of-the-art LEMON with 3.5X lower overhead than LEMON. Furthermore, FreeFuzz is able to detect 35 bugs for PyTorch and TensorFlow (with 31 confirmed by developers and 30 previously unknown).

Finding Deep-Learning Compilation Bugs with NNSmith

“…Metrics. We mainly target the following metrics for evaluation: • Code coverage: Following prior fuzzing work [7,8,59], we trace source-level branch coverage for both the entire systems and their pass-only components, measuring 1) total coverage counts all hit branches; and 2) unique coverage counts unique branches ("hard" branches) that other baselines cannot cover. • Bug counting: Following prior work [29,59,60], we use the number of independent patches as the number of detected bugs, except that we directly count the number of bug reports for closed-source systems (i.e., TensorRT) and unfixed ones.…”

“…There are several challenges facing the basic approach of fuzzing and differential testing, which are not addressed by prior work [33,57,59]. Next, we illustrate these challenges using concrete examples.…”

“…Finding DL compiler bugs requires generating input graphs that contain a variety of operators and connections. Some prior fuzzers [59] test only using single-operators and thus are too limiting. LEMON [57] and GraphFuzzer [33] generate multi-operator computation graphs, but they are restricted to certain types of operators and connections in order to avoid "type check" errors on the generated graphs (detailed in §6.1).…”

“…Therefore, prior techniques, such as Predoo [68], require users to manually set up the function arguments, and can only be evaluated on a limited number of APIs. More recently, FreeFuzz [59] aims to address this challenge via dynamically tracing API executions from various sources (including library documents, developer tests, and real-world models), and further mutates the traced inputs for each API to test DL libraries. While such API-level testing techniques are adequate for testing first-generation DL libraries ( §2.1), they can hardly find bugs in graph-level optimizations (e.g., 86% of the transformation bugs detected by NNSmith require multiple operators to trigger).…”

A Retrospective on ICSE 2022

“…The functionality of these equivalent graphs is verified to be identical. FreeFuzz takes the approach of mining usage of functions from open source code and carries out differential testing between similar concepts (e.g., CPU vs GPU computation should yield the same results) [33]. EAGLE and FreeFuzz are able to develop bugs, but another issue with debugging machine learning libraries is that these are frequently updated, and thus, the expected functionality of various methods may change over time.…”