Random testing for higher-order, stateful programs

Klein, Casey; Flatt, Matthew; Findler, Robert Bruce

doi:10.1145/1869459.1869505

Cited by 18 publications

(13 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Probably the work related most closely to ours is Klein et al [9], who generated random programs to test an objectoriented library, finding a large number of bugs. Their generator is capable of producing higher-order object-oriented programs (which override methods) and supports monitoring of pre-and post-conditions.…”

Section: Related Workmentioning

confidence: 91%

Testing an optimising compiler by generating random lambda terms

Pałka

Claessen

Russo

et al. 2011

Proceedings of the 6th International Workshop on Automation of Software Test

View full text Add to dashboard Cite

This paper considers random testing of a compiler, using randomly generated programs as inputs, and comparing their behaviour with and without optimisation. Since the generated programs must compile, then we need to take into account syntax, scope rules, and type checking during our random generation. Doing so, while attaining a good distribution of test data, proves surprisingly subtle; the main contribution of this paper is a workable solution to this problem. We used it to generate typed functions on lists, which we compiled using the Glasgow Haskell compiler, a mature production quality Haskell compiler. After around 20,000 tests we triggered an optimiser failure, and automatically simplified it to a program with just a few constructs.

show abstract

Section: Related Workmentioning

confidence: 91%

Testing an optimising compiler by generating random lambda terms

Pałka

Claessen

Russo

et al. 2011

Proceedings of the 6th International Workshop on Automation of Software Test

View full text Add to dashboard Cite

show abstract

“…Two test generators for Racket tests higher-order functions, either by generating new subclasses of existing classes, guided by the types of functions and the environment, and guided by developer-provided contracts [Klein et al 2010], or by adopting symbolic execution for higher-order functions [Nguyen and Horn 2015]. Another line of work generates functions to test a Haskell compiler [Palka et al 2011].…”

Section: Related Work 71 Test Generationmentioning

confidence: 99%

Test generation for higher-order functions in dynamic languages

Selakovic

Pradel

Karim³

et al. 2018

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

Test generation has proven to provide an effective way of identifying programming errors. Unfortunately, current test generation techniques are challenged by higher-order functions in dynamic languages, such as JavaScript functions that receive callbacks. In particular, existing test generators suffer from the unavailability of statically known type signatures, do not provide functions or provide only trivial functions as inputs, and ignore callbacks triggered by the code under test. This paper presents LambdaTester, a novel test generator that addresses the specific problems posed by higher-order functions in dynamic languages. The approach automatically infers at what argument position a method under test expects a callback, generates and iteratively improves callback functions given as input to this method, and uses novel test oracles that check whether and how callback functions are invoked. We apply LambdaTester to test 43 higher-order functions taken from 13 popular JavaScript libraries. The approach detects unexpected behavior in 12 of the 13 libraries, many of which are missed by a state-of-the-art test generator. CCS Concepts: • Software and its engineering → Software testing and debugging; Dynamic analysis;

show abstract

“…Heidegger and Thiemann (2010) use contracts to guide random testing for Javascript, allowing users to annotate inputs to combine different analyses for increasing the probability of hitting branches with highly constrained preconditions. Klein et al (2010) also extend random testing to work on higher-order stateful programs, discovering many bugs in object-oriented programs in Racket. Seidel et al (2015) use refinement types as generators for tests, significantly improving code coverage.…”

Section: Symbolic Executionmentioning

confidence: 99%

Higher order symbolic execution for contract verification and refutation

2016

View full text Add to dashboard Cite

We present a new approach to automated reasoning about higher-order programs by endowing symbolic execution with a notion of higher-order, symbolic values.To validate our approach, we use it to develop and evaluate a system for verifying and refuting behavioral software contracts of components in a functional language, which we call soft contract verification. In doing so, we discover a mutually beneficial relation between behavioral contracts and higher-order symbolic execution. Contracts aid symbolic execution by providing a rich language of specifications that can serve as the basis of symbolic higher-order values; the theory of blame enables modular verification and leads to the theorem that verified components can't be blamed; and the run-time monitoring of contracts enables soft verification whereby verified and unverified components can safely interact and verification is not an all-or-nothing proposition. Conversely, symbolic execution aids contracts by providing compile-time verification which increases assurance and enables optimizations; automated test-case generation for contracts with counter-examples; and engendering a virtuous cycle between verification and the gradual spread of contracts.Our system uses higher-order symbolic execution, leveraging contracts as a source of symbolic values including unknown behavioral values, and employs an updatable heap of contract invariants to reason about flow-sensitive facts. Whenever a contract is refuted, it reports a concrete counterexample reproducing the error, which may involve solving for an unknown function. The approach is able to analyze first-class contracts, recursive data structures, unknown functions, and control-flow-sensitive refinements of values, which are all idiomatic in dynamic languages. It makes effective use of an offthe-shelf solver to decide problems without heavy encodings. Our counterexample search is sound and relatively complete with respect to a first-order solver for base type values. Therefore, it can form the basis of automated verification and bug-finding tools for higher-order programs. The approach is competitive with a wide range of existing tools-including type systems, flow analyzers, and model checkers-on their own benchmarks. We have built a tool which analyzes programs written in Racket, and report on its effectiveness in verifying and refuting contracts. Higher-order symbolic execution for contract verification and refutation3 what a modular analysis means in this setting is tricky, but again contracts provide the answer. By re-using the concept of blame from Findler and Felleisen (2002), we define the errors that we rule out as exactly those that blame the portion of the program under consideration. This crucial distinction will become especially important when considering the behavior of unknown higher-order values.To demonstrate the first step in applying our approach, consider the following contrived, but illustrative example. Let positive? and negative? be predicates for positive and negative integers. Contracts ...

show abstract

Random testing for higher-order, stateful programs

Cited by 18 publications

References 25 publications

Testing an optimising compiler by generating random lambda terms

Testing an optimising compiler by generating random lambda terms

Test generation for higher-order functions in dynamic languages

Higher order symbolic execution for contract verification and refutation

Contact Info

Product

Resources

About