Muffin: Testing Deep Learning Libraries via Neural Architecture Fuzzing

Abstract

Deep learning (DL) techniques are proven effective in many challenging tasks, and become widely-adopted in practice. However, previous work has shown that DL libraries, the basis of building and executing DL models, contain bugs and can cause severe consequences. Unfortunately, existing testing approaches still cannot comprehensively exercise DL libraries. They utilize existing trained models and only detect bugs in model inference phase. In this work we propose Muffin to address these issues. To this end, Muffin applies a specifically-designed model fuzzing approach, which allows it to generate diverse DL models to explore the target library, instead of relying only on existing trained models. Muffin makes differential testing feasible in the model training phase by tailoring a set of metrics to measure the inconsistencies between different DL libraries. In this way, Muffin can best exercise the library code to detect more bugs. To evaluate the effectiveness of Muffin, we conduct experiments on three widely-used DL libraries. The results demonstrate that Muffin can detect 39 new bugs in the latest release versions of popular DL libraries, including Tensorflow, CNTK, and Theano. CCS Concepts: • Software and its engineering → Software testing and debugging; Software libraries and repositories.

NeuRI: Diversifying DNN Generation via Inductive Rule Inference

“…Meanwhile, there are two categories of proposals for constructing valid models. Weakly constrained model generation [13,23,42] limits the use of APIs to avoid dealing with fine-grained constraints. For example, LEMON [42] only operates shape-preserving operators without input constraints.…”

“…For example, LEMON [42] only operates shape-preserving operators without input constraints. More recent work [13,23] additionally inserts "reshaping" layers such that reshaped output tensors can stay in compatible shapes. However, it still may not construct operators with valid attributes (non-tensor arguments).…”

“…Built on Cradle, LEMON [42] and Audee [14] generate models via predefined mutation rules. Recently, Muffin [13] performs layer-by-layer model generation and tests both training and inference. More recently, NNSmith [21] annotates each operator with input constraints and shape transformation, and generates valid models aided by SMT solving.…”

“…More recently, NNSmith [21] annotates each operator with input constraints and shape transformation, and generates valid models aided by SMT solving. While they complement operator-level fuzzing, the model mutation/generation rules are restrictive, e.g., they typically only target naive shape-preserving operators [13,23,42], or require certain manual annotations [21], leading to a limited set of operators being used. This paper proposes to infer such operator rules automatically, and then leverage them for generating valid models with all possible operators.…”

Finding Deep-Learning Compilation Bugs with NNSmith

“…However, this design still substantially limits model diversity (as demonstrated in §2.3). The very recent (and concurrent) Muffin work [18] shares a similar limitation as GraphFuzzer: it uses "reshaping" layers to align tensor shapes during model generation; in addition, Muffin focuses on finding gradient computation bugs in DL libraries rather than DL compiler bugs. In this work, we aim to support more diverse/valid model generation for DL compiler fuzzing via a fundamentally different design powered by symbolic constraint solving [10] and gradient-driven search.…”