Muffin

Gu, Jiazhen; Luo, Xuchuan; Zhou, Yangfan; Wang, Xin

doi:10.1145/3510003.3510092

Cited by 19 publications

(31 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To evaluate the effectiveness of our approach, we compare our approach with four baselines, including two state-of-the-art approaches Muffin [15] and TVMFuzz [32]. Also, we evaluate them on three popular real-world deep learning compilers to investigate their bug detection capability.…”

Section: Discussionmentioning

confidence: 99%

“…3) Muffin: Muffin [15] is a state-of-the-art generation-based testing approach, proposed originally to test deep learning libraries such as Keras [33], generating models based on two model structure templates (chain structure with skips and cellbased structure) with deep learning library APIs. To satisfy with tensor structural constraints, Muffin hardcodes additional Reshaping layers to reshape the input/output tensors for the connection between layers.…”

Section: A Compared Workmentioning

confidence: 99%

“…For investigating bug detection capability, aligned with the setting in [15], for each method, we generate 300 computation graphs for testing compilers. Due to the difference of supported operations for each compiler, we run the generators with filtering the generated graphs that contain unsupported operations until the graph number reaches to 300.…”

Section: B Study Subjects and Settingsmentioning

confidence: 99%

“…To evaluate Isra, we implement it and empirically compare it with four baselines: (1) the aforementioned approach of random test generation, named as declarative-style generation, (2) test generation based on the idea of feedbackguided test generation [23], named Randoop-like generation, (3) a state-of-the-art tool named Muffin [15] implementing a generation-based approach for deep learning models, and (4) TVMFuzz [32], the only existing tool toward testing deep learning compilers. Our evaluation results show that our Isra approach substantially outperforms the baselines on the metrics of generated valid test inputs under the same setting, for demonstrating our approach's effectiveness.…”

Section: Introductionmentioning

confidence: 99%

“…• A domain-specific constraint solver toward finding solutions to the constraints specified from the semantic specifications of a deep learning model, with two novel strategies for resolving complex constraints. • Implementation and evaluation of our approach for showing its high effectiveness and high practical value, including outperforming state-of-the-art approaches (Muffin [15] and TVMFuzz [32]), and successfully finding 33 unique bugs in three popular real-world deep learning compilers.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Effective Random Test Generation for Deep Learning Compilers

Ren¹,

Wang²,

Xiong³

et al. 2023

Preprint

View full text Add to dashboard Cite

Deep learning compilers help address difficulties of deploying deep learning models on diverse types of hardware. Testing deep learning compilers is highly crucial, because they are impacting countless AI applications that use them for model optimization and deployment. To test deep learning compilers, random testing, being popularly used for compiler testing practices, faces the challenge of generating semantically valid test inputs, i.e., deep learning models that satisfy the semantic model specifications (in short as semantic specifications). To tackle this challenge, in this paper, we propose a novel approach named Isra, including a domain-specific constraint solver that resolves the constraints from the semantic specifications without backtracking. We implement and apply our approach on three popular real-world deep learning compilers including TVM, Glow, and a commercial compiler. The evaluation results show that Isra is more effective than the state-of-the-art approaches and the baseline approaches on constructing valid test inputs for compiler-bug detection, and Isra successfully finds 24 previously unknown bugs in released versions of the three compilers. These results indicate Isra's effectiveness and practical value.Index Terms-random testing, test generation, deep learning compilers, compiler testing, constraint solving 2 Checking against the semantic specifications can be in the form of a boolean checker such as repOK [2], etc.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: A Compared Workmentioning

confidence: 99%

Section: B Study Subjects and Settingsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Effective Random Test Generation for Deep Learning Compilers

Ren¹,

Wang²,

Xiong³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

TorchProbe: Fuzzing Dynamic Deep Learning Compilers

Su,

Geng,

Pekhimenko

et al. 2023

Programming Languages and Systems

View full text Add to dashboard Cite

Machine Learning-Based Fuzz Testing Techniques: A Survey

Zhang,

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Fuzz testing is a vulnerability discovery technique that tests the robustness of target programs by providing them with unconventional data. With the rapid increase in software quantity, scale and complexity, traditional fuzzing has revealed issues such as incomplete logic coverage, low automation level and insufficient test cases. Machine learning, with its exceptional capabilities in data analysis and classification prediction, presents a promising approach for improve fuzzing. This paper investigates the latest research results in fuzzing and provides a systematic review of machine learning-based fuzzing techniques. Firstly, by outlining the workflow of fuzzing, it summarizes the optimization of different stages of fuzzing using machine learning. Specifically, it focuses on the application of machine learning in the preprocessing phase, test case generation phase, input selection phase and result analysis phase. Secondly, it mentally focuses on the optimization methods of machine learning in the process of mutation, generation and filtering of test cases and compares and analyzes its technical principles. Furthermore, it analyzes the performance gains brought by applying machine learning techniques to fuzzing, mainly including coverage, vulnerability detection capability, efficiency and effectiveness of test cases. Lastly, it concludes by summarizing the challenges and difficulties in combining machine learning with fuzzing and presents prospects for future trends in this field.

show abstract

Muffin

Cited by 19 publications

References 44 publications

Effective Random Test Generation for Deep Learning Compilers

Effective Random Test Generation for Deep Learning Compilers

TorchProbe: Fuzzing Dynamic Deep Learning Compilers

Machine Learning-Based Fuzz Testing Techniques: A Survey

Contact Info

Product

Resources

About