We present FixML, a system for automatically generating feedback on logical errors in functional programming assignments. As functional languages have been gaining popularity, the number of students enrolling functional programming courses has increased significantly. However, the quality of feedback, in particular for logical errors, is hardly satisfying. To provide personalized feedback on logical errors, we present a new errorcorrection algorithm for functional languages, which combines statistical error-localization and type-directed program synthesis enhanced with components reduction and search space pruning using symbolic execution. We implemented our algorithm in a tool, called FixML, and evaluated it with 497 students' submissions from 13 exercises, including not only introductory but also more advanced problems. Our experimental results show that our tool effectively corrects various and complex errors: it fixed 43% of the 497 submissions in 5.4 seconds on average and managed to fix a hard-to-find error in a large submission, consisting of 154 lines. We also performed user study with 18 undergraduate students and confirmed that our system actually helps students to better understand their programming errors. CCS Concepts: • Software and its engineering → Automatic programming; Functional languages;
This is remarkable because those test cases have been carefully designed, refined, and used in the course over the last three years. Moreover, we found that the test cases generated by our technique are more concise and easier to understand the cause of logical errors than the manual test cases. Our experiments demonstrate that our approach is more effective and efficient than an existing property-based test case generator QCheck, an OCaml version of QuickCheck [Claessen and Hughes 2000], without any human effort. Furthermore, we show that our approach is useful in the context of automated program repair. When we used our counterexample generation algorithm in combination with an existing repair system for functional programs [Lee et al. 2018b], the number of test-suite-overfitted patches reduced significantly. Contributions. In this paper, we make the following contributions: • We propose a technique for detecting logical errors in functional programming assignments, which combines enumerative search and symbolic execution in a novel way. Our approach is fully automatic and is able to handle functional features such as higher-order functions effectively. • We conduct extensive evaluations with real students' submissions. The evaluation results demonstrate that our approach is effective both in error detection of real-world submissions and alleviating test-suite-overfitted patches in automated program repair. • We provide our counterexample algorithm as a tool, called TestML. Our tool and benchmarks used in the experiments are publicly available. 1 2 MOTIVATING EXAMPLES In this section, we motivate our technique with examples. We consider three programming exercises used in our undergraduate course on functional programming. Example 1. Let us consider a programming exercise, where students are asked to write a function, called diff, which symbolically differentiates arithmetic expressions. The arithmetic expressions are defined as an OCaml datatype as follows: type aexp = Const of int | Var of string | Power of (string * int) | Sum of aexp list | Times of aexp list ∆ ′ = unify(ϒ(l), int, ∆) new α int ⟨ l , ϒ, Γ, ∆⟩ ⟨α int , ∆ ′ (ϒ), ∆ ′ (Γ), ∆ ′ ⟩ E-Num ∆ ′ = unify(ϒ(l), string, ∆) new α string ⟨ l , ϒ, Γ, ∆⟩ ⟨α string , ∆ ′ (ϒ), ∆ ′ (Γ), ∆ ′ ⟩ E-Str c ∈ C Λ(c) = (τ 1 * τ 2) → T ∆ ′ = unify(ϒ(l),T , ∆) new l 1 , l 2 ⟨ l , ϒ, Γ, ∆⟩ ⟨(c(l 1 , l 2)), ∆ ′ (ϒ[l i → τ i ] 2 i=1), ∆ ′ (Γ[l i → Γ(l)] 2 i=1), ∆ ′ ⟩ E-Cnstr ∆ ′ = unify(ϒ(l), t 1 → t 2 , ∆) new t 1 , t 2 , l ′ ⟨ l , ϒ, Γ, ∆⟩ ⟨(λx. l ′), ∆ ′ (ϒ[l ′ → t 2 ]), ∆ ′ (Γ[l ′ → Γ(l)[x → t 1 ]]), ∆ ′ ⟩ E-Fun x ∈ dom(Γ(l)) ∆ ′ = unify(ϒ(l), Γ(l)(x), ∆) ⟨ l , ϒ, Γ, ∆⟩ ⟨x, ∆ ′ (ϒ), ∆ ′ (Γ), ∆ ′ ⟩ E-Var dom(Γ(l)) ∅ ∆ ′ = unify(ϒ(l), int, ∆) new l 1 , l 2 ⟨ l , ϒ, Γ, ∆⟩ ⟨(l 1 ⊕ l 2), ∆ ′ (ϒ[l 1 → int, l 2 → int]), ∆ ′ (Γ[l 1 → Γ(l), l 2 → Γ(l)]), ∆ ′ ⟩ E-Binop dom(Γ(l)) ∅ ∆ ′ = unify(ϒ(l), string, ∆) new l 1 , l 2 ⟨ l , ϒ, Γ, ∆⟩ ⟨(l 1ˆ l 2), ∆ ′ (ϒ[l 1 → string, l 2 → string]), ∆ ′ (Γ[l 1 → Γ(l), l 2 → Γ(l)]), ∆ ′ ⟩
No abstract
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.