Learning to blame: localizing novice type errors with data-driven diagnosis

Abstract

Localizing type errors is challenging in languages with global type inference, as the type checker must make assumptions about what the programmer intended to do. We introduce Nate, a data-driven approach to error localization based on supervised learning. Nate analyzes a large corpus of training data -pairs of ill-typed programs and their "fixed" versions -to automatically learn a model of where the error is most likely to be found. Given a new ill-typed program, Nate executes the model to generate a list of potential blame assignments ranked by likelihood. We evaluate Nate by comparing its precision to the state of the art on a set of over 5,000 ill-typed OCaml programs drawn from two instances of an introductory programming course. We show that when the top-ranked blame assignment is considered, Nate's data-driven model is able to correctly predict the exact sub-expression that should be changed 72% of the time, 28 points higher than OCaml and 16 points higher than the state-of-the-art SHErrLoc tool. Furthermore, Nate's accuracy surpasses 85% when we consider the top two locations and reaches 91% if we consider the top three.

Automatic diagnosis and correction of logical errors for functional programming assignments

“…For functional languages, prior works have focused on localizing and fixing type errors [Chen and Erwig 2014;Lerner et al 2007;Pavlinovic et al 2014Pavlinovic et al , 2015Seidel et al 2017;Zhang et al 2017]. For example, Seidel et al [2017] recently proposed a data-driven approach to localize type errors in OCaml programs. ; studied how type errors are fixed and presented Learnskell for producing user-friendly feedback on type errors.…”

“…Automated feedback generation. Recently, a large amount of work has been devoted to providing feedback on logical errors [D'Antoni et al 2016;Gulwani et al 2018;Kim et al 2016;Pu et al 2016;Singh et al 2013;Wang et al 2018], type errors [Seidel et al 2017;, syntax errors [Bhatia et al 2018;Gupta et al 2017], and performance problems [Gulwani et al 2014] in students' programs. AutoGrader [Singh et al 2013], which repairs students' programs using predetermined correction rules provided by instructors, has inspired recent advances in automatic feedback generation.…”

“…However, assisting students to resolve those errors is challenging. In particular, providing personalized feedback on logical errors was most difficult as it requires instructors to manually figure out the cause of the errors, unlike syntax or type errors for which a wealth of tool support already exists [Chen and Erwig 2014;Lerner et al 2007;Pavlinovic et al 2014Pavlinovic et al , 2015Seidel et al 2017;Zhang et al 2017].…”

“…While related existing work on functional languages has mostly focused on type errors (e.g. [Seidel et al 2017;Zhang et al 2017]), our system aims to provide personalized feedback on logical errors. • We present a novel error-correction algorithm that combines statistical error-localization and enhanced type-directed enumerative synthesis.…”

Gradual liquid type inference

“…Since the seminal work of Wand [1986], which keeps track of all unification steps in order to help debugging, many algorithms have been proposed to better assist programmers. A large variety of techniques has been explored, including slicing [Haack and Wells 2003;Tip and Dinesh 2001], heuristics [Zhang et al 2015], SMT constraint solving [Pavlinovic et al 2014], counter-example generation [Nguyen and Horn 2015;Seidel et al 2016], machine learning [Seidel et al 2017], and other search-based approaches, such as Seminal [Lerner et al 2007] and counterfactual change inference .…”