Testing probabilistic programming systems

Dutta, Saikat; Legunsen, Owolabi; Huang, Zixin; Misailovíc, Saša

doi:10.1145/3236024.3236057

Cited by 51 publications

(30 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Different from them, our work proposes a novel approach to testing DL libraries via effective model generation. Besides, there are some work focusing on testing machine learning (ML) libraries [26,27,57,58]. For example, Dwarakanath et al [27] adopted metamorphic testing to test ML libraries by conducting transformations on training and testing data.…”

Section: Related Workmentioning

confidence: 99%

Deep learning library testing via effective model generation

Wang

Yan

Chen

et al. 2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

View full text Add to dashboard Cite

Deep learning (DL) techniques are rapidly developed and have been widely adopted in practice. However, similar to traditional software, DL systems also contain bugs, which could cause serious impacts especially in safety-critical domains. Recently, much research has focused on testing DL models, while little attention has been paid for testing DL libraries, which is the basis of building DL models and directly affects the behavior of DL systems. In this work, we propose a novel approach, LEMON, to testing DL libraries. In particular, we (1) design a series of mutation rules for DL models, with the purpose of exploring different invoking sequences of library code and hard-to-trigger behaviors; and (2) propose a heuristic strategy to guide the model generation process towards the direction of amplifying the inconsistent degrees of the inconsistencies between different DL libraries caused by bugs, so as to mitigate the impact of potential noise introduced by uncertain factors in DL libraries. We conducted an empirical study to evaluate the effectiveness of LEMON with 20 release versions of 4 widely-used DL libraries, i.e., TensorFlow, Theano, CNTK, MXNet. The results demonstrate that LEMON detected 24 new bugs in the latest release versions of these libraries, where 7 bugs have been confirmed and one bug has been fixed by developers. Besides, the results confirm that the heuristic strategy for model generation indeed effectively guides LEMON in amplifying the inconsistent degrees for bugs.

show abstract

Section: Related Workmentioning

confidence: 99%

Deep learning library testing via effective model generation

Wang

Yan

Chen

et al. 2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

View full text Add to dashboard Cite

show abstract

“…Storm-IR is general enough to represent the core of majority of the example programs included in the repositories of these languages and allows the translators to handle the language-specific features discussed above. Our intermediate language draws inspiration from Probfuzz [10], but improves expressivity and generality (e.g., it allows arbitrary inter-leavings of statements like sampling, assignment, observes and loops). This allows the Storm-IR to represent a richer and more diverse set of probabilistic programs used across different PP systems.…”

Section: Translators and Storm-irmentioning

confidence: 99%

“…The numerical and approximate nature of PP systems and implementation complexity make it hard to ensure their correctness, and subtle bugs can easily remain unnoticed [20,38,47]. Our recent study [10] showed that over 25% of all bugs in three popular systems are domain specific, including algorithmic, numerical, boundary condition, dimensional, and accuracy bugs. The bugs manifest as wrong results, crashes, infinite loops, or numerical exceptions.…”

Section: Introductionmentioning

confidence: 99%

“…We use the reducer to generate smaller programs that reveal failures in two state-of-the-art PP systems: Stan [8,15,29], one of the most mature and frequently-used PP systems, and Pyro [4], a Python-based deep probabilistic modeling framework from Uber. We studied three sources of bugs: 1) a probabilistic bug database we created in previous research [10], which includes test programs that were already minimized by a human, 2) new bugs discovered using ProbFuzz [10], and 3) a repository with representative Stan models that offers larger probabilistic programs [48]. In total, we analyzed 47 programs (34 from Stan and 13 from Pyro).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Storm: program reduction for testing and debugging probabilistic programming systems

Dutta

Zhang

Huang

et al. 2019

Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of

Self Cite

View full text Add to dashboard Cite

Probabilistic programming languages offer an intuitive way to model uncertainty by representing complex probability models as simple probabilistic programs. Probabilistic programming systems (PP systems) hide the complexity of inference algorithms away from the program developer. Unfortunately, if a failure occurs during the run of a PP system, a developer typically has very little support in finding the part of the probabilistic program that causes the failure in the system. This paper presents Storm, a novel general framework for reducing probabilistic programs. Given a probabilistic program (with associated data and inference arguments) that causes a failure in a PP system, Storm finds a smaller version of the program, data, and arguments that cause the same failure. Storm leverages both generic code and data transformations from compiler testing and domain-specific, probabilistic transformations. The paper presents new transformations that reduce the complexity of statements and expressions, reduce data size, and simplify inference arguments (e.g., the number of iterations of the inference algorithm). We evaluated Storm on 47 programs that caused failures in two popular probabilistic programming systems, Stan and Pyro. Our experimental results show Storm's effectiveness. For Stan, our minimized programs have 49% less code, 67% less data, and 96% fewer iterations. For Pyro, our minimized programs have 58% less code, 96% less data, and 99% fewer iterations. We also show the benefits of Storm when debugging probabilistic programs. CCS CONCEPTS • Software and its engineering → Software testing and debugging.

show abstract

“…It is well-known that floating-point computation can be inaccurate due to the finite representation of floating-point numbers, and inaccuracies can lead to catastrophes, such as stock market disorder [Quinn 1983], incorrect election results [Weber-Wulff 1992], rocket launch failure [Lions et al 1996], and the loss of human lives [Skeel 1992]. Modern systems also suffer from numerical inaccuracies, such as probabilistic programming systems [Dutta et al 2018] and deep learning libraries [Pham et al 2019]. The increased complexity of modern systems makes it even more important and challenging to detect floating-point errors.…”

Section: Introductionmentioning

confidence: 99%

Detecting floating-point errors via atomic conditions

Zou

Zeng

Xiong

et al. 2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

This paper tackles the important, difficult problem of detecting program inputs that trigger large floating-point errors in numerical code. It introduces a novel, principled dynamic analysis that leverages the mathematically rigorously analyzed condition numbers for atomic numerical operations, which we call atomic conditions, to effectively guide the search for large floating-point errors. Compared with existing approaches, our work based on atomic conditions has several distinctive benefits: (1) it does not rely on high-precision implementations to act as approximate oracles, which are difficult to obtain in general and computationally costly; and (2) atomic conditions provide accurate, modular search guidance. These benefits in combination lead to a highly effective approach that detects more significant errors in real-world code (e.g., widely-used numerical library functions) and achieves several orders of speedups over the state-of-the-art, thus making error analysis significantly more practical. We expect the methodology and principles behind our approach to benefit other floating-point program analysis tasks such as debugging, repair and synthesis. To facilitate the reproduction of our work, we have made our implementation, evaluation data and results publicly available on GitHub at https://github.com/FP-Analysis/atomic-condition. CCS Concepts: • Software and its engineering → General programming languages; Software testing and debugging.

show abstract

Testing probabilistic programming systems

Cited by 51 publications

References 46 publications

Deep learning library testing via effective model generation

Deep learning library testing via effective model generation

Storm: program reduction for testing and debugging probabilistic programming systems

Detecting floating-point errors via atomic conditions

Contact Info

Product

Resources

About