Automatic Testing of Higher Order Functions

Koopman, Pieter; Plasmeijer, Rinus

doi:10.1007/11924661_9

Cited by 12 publications

(10 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To design a test generator and an input shrinker for functional test cases manually is challenging as they require programmers to design grammar for an input function. Some researches focus on addressing the problem of testing higher-order function in property-based testing (e.g., [Koopman and Plasmeijer 2006]).…”

Section: Comparison With Property-based Testingmentioning

confidence: 99%

“…It, however, has a difficulty in generating function type test cases and requires users to build test generators manually. To address these limitations, several researches have been proposed [Koopman and Plasmeijer 2006;Lampropoulos et al 2017;Löscher and Sagonas 2017]. Koopman and Plasmeijer [2006] improved property-based testing to test higher-order function more easily by representing functions' AST as a data type format and generating instances of this data type using property-based testing tool.…”

Section: Related Workmentioning

confidence: 99%

“…To address these limitations, several researches have been proposed [Koopman and Plasmeijer 2006;Lampropoulos et al 2017;Löscher and Sagonas 2017]. Koopman and Plasmeijer [2006] improved property-based testing to test higher-order function more easily by representing functions' AST as a data type format and generating instances of this data type using property-based testing tool. As constructing a test case generator is a burdensome task for users, Lampropoulos et al [2017] demonstrated Luck, a language which allows users to easily build, read, and maintain the property-based test generators.…”

Section: Related Workmentioning

confidence: 99%

“…Higher-order Function Testing. Several researches have been presented to test higher-order function [Heidegger and Thiemann 2010;Klein et al 2010;Koopman and Plasmeijer 2006;Selakovic et al 2018]. Klein et al [2010] and Heidegger and Thiemann [2010] extended random testing to work on higher-order program.…”

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

Automatic and scalable detection of logical errors in functional programming assignments

Song

Lee

2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

This is remarkable because those test cases have been carefully designed, refined, and used in the course over the last three years. Moreover, we found that the test cases generated by our technique are more concise and easier to understand the cause of logical errors than the manual test cases. Our experiments demonstrate that our approach is more effective and efficient than an existing property-based test case generator QCheck, an OCaml version of QuickCheck [Claessen and Hughes 2000], without any human effort. Furthermore, we show that our approach is useful in the context of automated program repair. When we used our counterexample generation algorithm in combination with an existing repair system for functional programs [Lee et al. 2018b], the number of test-suite-overfitted patches reduced significantly. Contributions. In this paper, we make the following contributions: • We propose a technique for detecting logical errors in functional programming assignments, which combines enumerative search and symbolic execution in a novel way. Our approach is fully automatic and is able to handle functional features such as higher-order functions effectively. • We conduct extensive evaluations with real students' submissions. The evaluation results demonstrate that our approach is effective both in error detection of real-world submissions and alleviating test-suite-overfitted patches in automated program repair. • We provide our counterexample algorithm as a tool, called TestML. Our tool and benchmarks used in the experiments are publicly available. 1 2 MOTIVATING EXAMPLES In this section, we motivate our technique with examples. We consider three programming exercises used in our undergraduate course on functional programming. Example 1. Let us consider a programming exercise, where students are asked to write a function, called diff, which symbolically differentiates arithmetic expressions. The arithmetic expressions are defined as an OCaml datatype as follows: type aexp = Const of int | Var of string | Power of (string * int) | Sum of aexp list | Times of aexp list ∆ ′ = unify(ϒ(l), int, ∆) new α int ⟨ l , ϒ, Γ, ∆⟩ ⟨α int , ∆ ′ (ϒ), ∆ ′ (Γ), ∆ ′ ⟩ E-Num ∆ ′ = unify(ϒ(l), string, ∆) new α string ⟨ l , ϒ, Γ, ∆⟩ ⟨α string , ∆ ′ (ϒ), ∆ ′ (Γ), ∆ ′ ⟩ E-Str c ∈ C Λ(c) = (τ 1 * τ 2) → T ∆ ′ = unify(ϒ(l),T , ∆) new l 1 , l 2 ⟨ l , ϒ, Γ, ∆⟩ ⟨(c(l 1 , l 2)), ∆ ′ (ϒ[l i → τ i ] 2 i=1), ∆ ′ (Γ[l i → Γ(l)] 2 i=1), ∆ ′ ⟩ E-Cnstr ∆ ′ = unify(ϒ(l), t 1 → t 2 , ∆) new t 1 , t 2 , l ′ ⟨ l , ϒ, Γ, ∆⟩ ⟨(λx. l ′), ∆ ′ (ϒ[l ′ → t 2 ]), ∆ ′ (Γ[l ′ → Γ(l)[x → t 1 ]]), ∆ ′ ⟩ E-Fun x ∈ dom(Γ(l)) ∆ ′ = unify(ϒ(l), Γ(l)(x), ∆) ⟨ l , ϒ, Γ, ∆⟩ ⟨x, ∆ ′ (ϒ), ∆ ′ (Γ), ∆ ′ ⟩ E-Var dom(Γ(l)) ∅ ∆ ′ = unify(ϒ(l), int, ∆) new l 1 , l 2 ⟨ l , ϒ, Γ, ∆⟩ ⟨(l 1 ⊕ l 2), ∆ ′ (ϒ[l 1 → int, l 2 → int]), ∆ ′ (Γ[l 1 → Γ(l), l 2 → Γ(l)]), ∆ ′ ⟩ E-Binop dom(Γ(l)) ∅ ∆ ′ = unify(ϒ(l), string, ∆) new l 1 , l 2 ⟨ l , ϒ, Γ, ∆⟩ ⟨(l 1ˆ l 2), ∆ ′ (ϒ[l 1 → string, l 2 → string]), ∆ ′ (Γ[l 1 → Γ(l), l 2 → Γ(l)]), ∆ ′ ⟩

show abstract

Section: Comparison With Property-based Testingmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Automatic and scalable detection of logical errors in functional programming assignments

Song

Lee

2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

show abstract

“…However, the generated functions do not modify any other state beyond the return value. Koopman and Plasmeijer propose to improve QuickCheck by systematically generating functions based on the AST representation of a function argument [Koopman and Plasmeijer 2006]. The basic idea of their approach is to represent functions as a data type and to systematically enumerate elements of this data type.…”

Section: Related Work 71 Test Generationmentioning

confidence: 99%

Test generation for higher-order functions in dynamic languages

Selakovic

Pradel

Karim³

et al. 2018

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

Test generation has proven to provide an effective way of identifying programming errors. Unfortunately, current test generation techniques are challenged by higher-order functions in dynamic languages, such as JavaScript functions that receive callbacks. In particular, existing test generators suffer from the unavailability of statically known type signatures, do not provide functions or provide only trivial functions as inputs, and ignore callbacks triggered by the code under test. This paper presents LambdaTester, a novel test generator that addresses the specific problems posed by higher-order functions in dynamic languages. The approach automatically infers at what argument position a method under test expects a callback, generates and iteratively improves callback functions given as input to this method, and uses novel test oracles that check whether and how callback functions are invoked. We apply LambdaTester to test 43 higher-order functions taken from 13 popular JavaScript libraries. The approach detects unexpected behavior in 12 of the 13 libraries, many of which are missed by a state-of-the-art test generator. CCS Concepts: • Software and its engineering → Software testing and debugging; Dynamic analysis;

show abstract

Sound and Complete Concolic Testing for Higher-order Functions

You

Findler

Dimoulas

2021

Programming Languages and Systems

View full text Add to dashboard Cite

Higher-order functions have become a staple of modern programming languages. However, such values stymie concolic testers, as the SMT solvers at their hearts are inherently first-order.This paper lays a formal foundations for concolic testing higher-order functional programs. Three ideas enable our results: (i) our tester considers only program inputs in a canonical form; (ii) it collects novel constraints from the evaluation of the canonical inputs to search the space of inputs with partial help from an SMT solver and (iii) it collects constraints from canonical inputs even when they are arguments to concretized calls. We prove that (i) concolic evaluation is sound with respect to concrete evaluation; (ii) modulo concretization and SMT solver incompleteness, the search for a counter-example succeeds if a user program has a bug and (iii) this search amounts to directed evolution of inputs targeting hard-to-reach corners of the program.

show abstract

Automatic Testing of Higher Order Functions

Cited by 12 publications

References 8 publications

Automatic and scalable detection of logical errors in functional programming assignments

Automatic and scalable detection of logical errors in functional programming assignments

Test generation for higher-order functions in dynamic languages

Sound and Complete Concolic Testing for Higher-order Functions

Contact Info

Product

Resources

About