Nagini: A Static Verifier for Python

Eilers, Marco; Müller, Péter

doi:10.1007/978-3-319-96145-3_33

Cited by 35 publications

(27 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, the verification technique must be usable by scientists with little to no training in computer science. The state of the art techniques require a deep knowledge of formal methods to use them effectively [13]. Second, the amount of time spent verifying the software must be small compared to the amount of time spent writing the code to be verified.…”

Section: Discussionmentioning

confidence: 99%

Refinement Type Contracts for Verification of Scientific Investigative Software

Shinn

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Our scientific knowledge is increasingly built on software output. User code which defines data analysis pipelines and computational models is essential for research in the natural and social sciences, but little is known about how to ensure its correctness. The structure of this code and the development process used to build it limit the utility of traditional testing methodology. Formal methods for software verification have seen great success in ensuring code correctness but generally require more specialized training, development time, and funding than is available in the natural and social sciences. Here, we present a Python library which uses lightweight formal methods to provide correctness guarantees without the need for specialized knowledge or substantial time investment. Our package provides runtime verification of function entry and exit condition contracts using refinement types. It allows checking hyperproperties within contracts and offers automated test case generation to supplement online checking. We co-developed our tool with a mediumsized (≈3000 LOC) software package which simulates decision-making in cognitive neuroscience. In addition to helping us locate trivial bugs earlier on in the development cycle, our tool was able to locate four bugs which may have been difficult to find using traditional testing methods. It was also able to find bugs in user code which did not contain contracts or refinement type annotations. This demonstrates how formal methods can be used to verify the correctness of scientific software which is difficult to test with mainstream approaches.

show abstract

Section: Discussionmentioning

confidence: 99%

Refinement Type Contracts for Verification of Scientific Investigative Software

Shinn

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…To the best of our knowledge, there is no existing work on Runtime Verification of web services. We are also unaware of other (available and maintained) RV tools for Python (there is Nagini [17], but this focuses on static verification) as most either operate offline (on log files) or focus on other languages such as Java [5,7,18] using AspectJ for instrumentation, C [19], or Erlang [20]. Few RV tools consider the instrumentation problem within the tool.…”

Section: Related Workmentioning

confidence: 99%

VyPR2: A Framework for Runtime Verification of Python Web Services

Dawes

Reger

Franzoni

et al. 2019

Tools and Algorithms for the Construction and Analysis of Systems

View full text Add to dashboard Cite

Runtime Verification (RV) is the process of checking whether a run of a system holds a given property. In order to perform such a check online, the algorithm used to monitor the property must induce minimal overhead. This paper focuses on two areas that have received little attention from the RV community: Python programs and web services. Our first contribution is the VyPR runtime verification tool for singlethreaded Python programs. The tool handles specifications in our, previously introduced, Control-Flow Temporal Logic (CFTL), which supports the specification of state and time constraints over runs of functions. VyPR minimally (in terms of reachability) instruments the input program with respect to a CFTL specification and then uses instrumentation information to optimise the monitoring algorithm. Our second contribution is the lifting of VyPR to the web service setting, resulting in the VyPR2 tool. We first describe the necessary modifications to the architecture of VyPR, and then describe our experience applying VyPR2 to a service that is critical to the physics reconstruction pipeline on the CMS Experiment at CERN.

show abstract

“…In order to demonstrate that the Axiom Profiler can be used to identify and help explain matching loops in real world problems, we ran experiments on a total of 34,159 SMT files, corresponding to the full test suites of a selection of verification tools which use Z3 as a backend: F* [22] (6,281 files), Why3 [10] (26,258 files), Viper (887 files) [18], Nagini [9] (266 files) and Dafny [13] (467 files)). Using a command-line interface for our tool, we analysed each of these files in an attempt to find matching loops.…”

Section: Implementation and Evaluationmentioning

confidence: 99%

The Axiom Profiler: Understanding and Debugging SMT Quantifier Instantiations

Becker

Müller

Summers

2019

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

SMT solvers typically reason about universal quantifiers via E-matching: syntactic matching patterns for each quantifier prescribe shapes of ground terms whose presence in the SMT run will trigger quantifier instantiations. The effectiveness and performance of the SMT solver depend crucially on well-chosen patterns. Overly restrictive patterns cause relevant quantifier instantiations to be missed, while overly permissive patterns can cause performance degradation including nontermination if the solver gets stuck in a matching loop. Understanding and debugging such instantiation problems is an overwhelming task, due to the typically large number of quantifier instantiations and their non-trivial interactions with each other and other solver aspects. In this paper, we present the Axiom Profiler, a tool that enables users to analyse instantiation problems effectively, by filtering and visualising rich logging information from SMT runs. Our tool implements novel techniques for automatically detecting matching loops and explaining why they repeat indefinitely. We evaluated the tool on the full test suites of five existing program verifiers, where it discovered and explained multiple previouslyunknown matching loops.

show abstract

Nagini: A Static Verifier for Python

Cited by 35 publications

References 18 publications

Refinement Type Contracts for Verification of Scientific Investigative Software

Refinement Type Contracts for Verification of Scientific Investigative Software

VyPR2: A Framework for Runtime Verification of Python Web Services

The Axiom Profiler: Understanding and Debugging SMT Quantifier Instantiations

Contact Info

Product

Resources

About