Generating good generators for inductive relations

Lampropoulos, Leonidas; Paraskevopoulou, Zoe; Pierce, Benjamin C.

doi:10.1145/3158133

Cited by 30 publications

(9 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Regardless of the input node, Tzer simply returns a new node generated from scratch that satisfies the given constraints. This is done by Tzer's generator, which is inspired by the prior generators in the random testing community [Claessen et al 2015;Lampropoulos et al 2017]. The functionality of the generator is to produce IR ingredients/snippets based on the constraints and a Deletion.…”

Section: Anynodetype =mentioning

confidence: 99%

Coverage-Guided Tensor Compiler Fuzzing with Joint IR-Pass Mutation

Liu¹,

Wei²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

In the past decade, Deep Learning (DL) systems have been widely deployed in various application domains to facilitate our daily life, e.g., natural language processing, healthcare, activity recognition, and autonomous driving. Meanwhile, it is extremely challenging to ensure the correctness of DL systems (e.g., due to their intrinsic nondeterminism), and bugs in DL systems can cause serious consequences and may even threaten human lives. In the literature, researchers have explored various techniques to test, analyze, and verify DL models, since their quality directly affects the corresponding system behaviors. Recently, researchers have also proposed novel techniques for testing the underlying operator-level DL libraries (such as TensorFlow and PyTorch), which provide general binary implementations for each high-level DL operator and are the foundation for running DL models on different hardware platforms. However, there is still limited work targeting the reliability of the emerging tensor compilers (also known as DL compilers), which aim to automatically compile high-level tensor computation graphs directly into high-performance binaries for better efficiency, portability, and scalability than traditional operator-level libraries. Therefore, in this paper, we target the important problem of tensor compiler testing, and have proposed Tzer, a practical fuzzing technique for the widely used TVM tensor compiler. Tzer focuses on mutating the low-level Intermediate Representation (IR) for TVM due to the limited mutation space for the high-level IR. More specifically, Tzer leverages both general-purpose and tensor-compiler-specific mutators guided by coverage feedback for diverse and evolutionary IR mutation; furthermore, since tensor compilers provide various passes (i.e., transformations) for IR optimization, Tzer also performs pass mutation in tandem with IR mutation for more effective fuzzing. Our experimental results show that Tzer substantially outperforms existing fuzzing techniques on tensor compiler testing, with 75% higher coverage and 50% more valuable tests than the 2nd-best technique. Also, different components of Tzer have been validated via ablation study. To date, Tzer has detected 49 previously unknown bugs for TVM, with 37 bugs confirmed and 25 bugs fixed (PR merged).

show abstract

Section: Anynodetype =mentioning

confidence: 99%

Coverage-Guided Tensor Compiler Fuzzing with Joint IR-Pass Mutation

Liu¹,

Wei²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Hammers and other general-purpose automation (Section 5.2.1) can help a proof engineer discharge simple proof obligations and quickly determine that a theorem is true; the proof engineer can then reprove the theroem in a different way if desired. Propertybased testing tools like Quickcheck for Isabelle (Bulwahn, 2012) and QuickChick (Lampropoulos et al, 2017;Paraskevopoulou et al, 2015) for Coq can help users identify counterexamples to false properties.…”

Section: Tooling For User Supportmentioning

confidence: 99%

QED at Large: A Survey of Engineering of Formally Verified Software

Ringer

Palmskog

Sergey

et al. 2019

FNT in Programming Languages

View full text Add to dashboard Cite

Development of formal proofs of correctness of programs can increase actual and perceived reliability and facilitate better understanding of program specifications and their underlying assumptions. Tools supporting such development have been available for over 40 years, but have only recently seen wide practical use. Projects based on construction of machine-checked formal proofs are now reaching an unprecedented scale, comparable to large software projects, which leads to new challenges in proof development and maintenance. Despite its increasing importance, the field of proof engineering is seldom considered in its own right; related theories, techniques, and tools span many fields and venues. This survey of the literature presents a holistic understanding of proof engineering for program correctness, covering impact in practice, foundations, proof automation, proof organization, and practical proof development.

show abstract

“…There are tools for random testing such as QuickCheck [Claessen and Hughes 2000], Smallcheck [Runciman et al 2008], and QuickChick [Lampropoulos et al 2018] that use type information and additional properties to generate random tests. However, we are not aware that these tools have been used to generate worst-case inputs or tests for exposing high resource usage.…”

Section: Related Workmentioning

confidence: 99%

Type-guided worst-case input generation

Wang

Hoffmann

2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

This paper presents a novel technique for type-guided worst-case input generation for functional programs. The technique builds on automatic amortized resource analysis (AARA), a type-based technique for deriving symbolic bounds on the resource usage of functions. Worst-case input generation is performed by an algorithm that takes as input a function, its resource-annotated type derivation in AARA, and a skeleton that describes the shape and size of the input that is to be generated. If successful, the algorithm fills in integers, booleans, and data structures to produce a value of the shape given by the skeleton. The soundness theorem states that the generated value exhibits the highest cost among all arguments of the functions that have the shape of the skeleton. This cost corresponds exactly to the worst-case bound that is established by the type derivation. In this way, a successful completion of the algorithm proves that the bound is tight for inputs of the given shape. Correspondingly, a relative completeness theorem is proved to show that the algorithm succeeds if and only if the derived worst-case bound is tight. The theorem is relative because it depends on a decision procedure for constraint solving. The technical development is presented for a simple first-order language with linear resource bounds. However, the technique scales to and has been implemented for Resource Aware ML, an implementation of AARA for a fragment of OCaml with higher-order functions, user-defined data types, and types for polynomial bounds. Experiments demonstrate that the technique works effectively and can derive worst-case inputs with hundreds of integers for sorting algorithms, operations on search trees, and insertions into hash tables.

show abstract

Generating good generators for inductive relations

Cited by 30 publications

References 32 publications

Coverage-Guided Tensor Compiler Fuzzing with Joint IR-Pass Mutation

Coverage-Guided Tensor Compiler Fuzzing with Joint IR-Pass Mutation

QED at Large: A Survey of Engineering of Formally Verified Software

Type-guided worst-case input generation

Contact Info

Product

Resources

About