Automatic diagnosis and correction of logical errors for functional programming assignments

Lee, Jun-Ho; Song, Dowon; So, Sunbeom; Oh, Hakjoo

doi:10.1145/3276528

Cited by 16 publications

(12 citation statements)

References 44 publications

(71 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Experimental Setting. For evaluation, we implemented a counter-example guided repair system based on FixML [Lee et al 2018b], a state-of-the-art feedback generator for functional programming assignments. As FixML requires a reference implementation to effectively repair a buggy program, we provided a solution program implemented by an instructor as a reference code.…”

Section: Enhancing Automatic Program Repair With Automatic Test Genermentioning

confidence: 99%

“…Automatic Program Repair. Recently, automatic program repair technique has been widely used to fix bugs in general programs Kim et al 2013;Le Goues et al 2012;Long and Rinard 2016;Nguyen et al 2013; or students' implementations [Bhatia et al 2018;D'Antoni et al 2016;Gupta et al 2017;Lee et al 2018b;Pu et al 2016;Singh et al 2013]. Most of these works use a test suite as a specification of the generated patches.…”

Section: Related Workmentioning

confidence: 99%

“…Program Synthesis. Program synthesis technique has been widely used in various domains such as generation of complex APIs [Feng et al 2017] or SQL queries Yaghmazadeh et al 2017], and programming education [D'Antoni et al 2016;Lee et al 2018b;Pu et al 2016;Singh et al 2013;Oh 2017, 2018]. As the program synthesis gains the popularity recently, a lot of researches are proposed to improve the performance of it.…”

Section: Related Workmentioning

confidence: 99%

“…Machine learning technique such as deep learning [Balog et al 2017] or probabilistic model [Lee et al 2018a] were used to enhance program synthesis by giving priority to candidates which are likely to be solutions. Applying static analysis or dynamic symbolic execution is another promising approach for pruning out the redundant states during program synthesis [Lee et al 2018b;Oh 2017, 2018]. To generate a well-typed test case and to prove whether it is an actual counter-example, we adopted two techniques above, type-directed search and SMT solving based on symbolic execution.…”

Section: Related Workmentioning

confidence: 99%

“…However, these systems often produce test-suite-overfitted patches which retain some bugs but satisfy the given test cases. We show that our technique can enhance the performance of an existing program repair system, FixML [Lee et al 2018b], for functional programming assignments.…”

Section: Introductionmentioning

confidence: 96%

See 4 more Smart Citations

Automatic and scalable detection of logical errors in functional programming assignments

Song

Lee

2019

Proc. ACM Program. Lang.

Self Cite

View full text Add to dashboard Cite

This is remarkable because those test cases have been carefully designed, refined, and used in the course over the last three years. Moreover, we found that the test cases generated by our technique are more concise and easier to understand the cause of logical errors than the manual test cases. Our experiments demonstrate that our approach is more effective and efficient than an existing property-based test case generator QCheck, an OCaml version of QuickCheck [Claessen and Hughes 2000], without any human effort. Furthermore, we show that our approach is useful in the context of automated program repair. When we used our counterexample generation algorithm in combination with an existing repair system for functional programs [Lee et al. 2018b], the number of test-suite-overfitted patches reduced significantly. Contributions. In this paper, we make the following contributions: • We propose a technique for detecting logical errors in functional programming assignments, which combines enumerative search and symbolic execution in a novel way. Our approach is fully automatic and is able to handle functional features such as higher-order functions effectively. • We conduct extensive evaluations with real students' submissions. The evaluation results demonstrate that our approach is effective both in error detection of real-world submissions and alleviating test-suite-overfitted patches in automated program repair. • We provide our counterexample algorithm as a tool, called TestML. Our tool and benchmarks used in the experiments are publicly available. 1 2 MOTIVATING EXAMPLES In this section, we motivate our technique with examples. We consider three programming exercises used in our undergraduate course on functional programming. Example 1. Let us consider a programming exercise, where students are asked to write a function, called diff, which symbolically differentiates arithmetic expressions. The arithmetic expressions are defined as an OCaml datatype as follows: type aexp = Const of int | Var of string | Power of (string * int) | Sum of aexp list | Times of aexp list ∆ ′ = unify(ϒ(l), int, ∆) new α int ⟨ l , ϒ, Γ, ∆⟩ ⟨α int , ∆ ′ (ϒ), ∆ ′ (Γ), ∆ ′ ⟩ E-Num ∆ ′ = unify(ϒ(l), string, ∆) new α string ⟨ l , ϒ, Γ, ∆⟩ ⟨α string , ∆ ′ (ϒ), ∆ ′ (Γ), ∆ ′ ⟩ E-Str c ∈ C Λ(c) = (τ 1 * τ 2) → T ∆ ′ = unify(ϒ(l),T , ∆) new l 1 , l 2 ⟨ l , ϒ, Γ, ∆⟩ ⟨(c(l 1 , l 2)), ∆ ′ (ϒ[l i → τ i ] 2 i=1), ∆ ′ (Γ[l i → Γ(l)] 2 i=1), ∆ ′ ⟩ E-Cnstr ∆ ′ = unify(ϒ(l), t 1 → t 2 , ∆) new t 1 , t 2 , l ′ ⟨ l , ϒ, Γ, ∆⟩ ⟨(λx. l ′), ∆ ′ (ϒ[l ′ → t 2 ]), ∆ ′ (Γ[l ′ → Γ(l)[x → t 1 ]]), ∆ ′ ⟩ E-Fun x ∈ dom(Γ(l)) ∆ ′ = unify(ϒ(l), Γ(l)(x), ∆) ⟨ l , ϒ, Γ, ∆⟩ ⟨x, ∆ ′ (ϒ), ∆ ′ (Γ), ∆ ′ ⟩ E-Var dom(Γ(l)) ∅ ∆ ′ = unify(ϒ(l), int, ∆) new l 1 , l 2 ⟨ l , ϒ, Γ, ∆⟩ ⟨(l 1 ⊕ l 2), ∆ ′ (ϒ[l 1 → int, l 2 → int]), ∆ ′ (Γ[l 1 → Γ(l), l 2 → Γ(l)]), ∆ ′ ⟩ E-Binop dom(Γ(l)) ∅ ∆ ′ = unify(ϒ(l), string, ∆) new l 1 , l 2 ⟨ l , ϒ, Γ, ∆⟩ ⟨(l 1ˆ l 2), ∆ ′ (ϒ[l 1 → string, l 2 → string]), ∆ ′ (Γ[l 1 → Γ(l), l 2 → Γ(l)]), ∆ ′ ⟩

show abstract

Section: Enhancing Automatic Program Repair With Automatic Test Genermentioning

confidence: 99%