Neurosymbolic repair for low-code formula languages

Abstract

Most users of low-code platforms, such as Excel and PowerApps, write programs in domain-specific formula languages to carry out nontrivial tasks. Often users can write most of the program they want, but introduce small mistakes that yield broken formulas. These mistakes, which can be both syntactic and semantic, are hard for low-code users to identify and fix, even though they can be resolved with just a few edits. We formalize the problem of producing such edits as the last-mile repair problem. To address this problem, we developed LaMirage, a LAst-MIle RepAir-engine GEnerator that combines symbolic and neural techniques to perform last-mile repair in low-code formula languages. LaMirage takes a grammar and a set of domain-specific constraints/rules, which jointly approximate the target language, and uses these to generate a repair engine that can fix formulas in that language. To tackle the challenges of localizing errors and ranking candidate repairs, LaMirage leverages neural techniques, whereas it relies on symbolic methods to generate candidate edits. This combination allows LaMirage to find repairs that satisfy the provided grammar and constraints, and then pick the most natural repair. We compare LaMirage to state-of-the-art neural and symbolic approaches on 400 real Excel and Power Fx formulas, where LaMirage outperforms all baselines. We release these benchmarks to encourage subsequent work in low-code domains.

FLAME: A Small Language Model for Spreadsheet Formulas

“…We evaluate FLAME's ability to perform last-mile formula repair (Bavishi et al 2022), completion, and similar formula retrieval. Prior work in the repair domain includes Deep-Fix (Gupta et al 2017), BIFI (Yasunaga and Liang 2021), Dr.Repair (Yasunaga and Liang 2020), TFix (Berabi et al 2021), and RING (Joshi et al 2022), which use deep learning to perform syntax, compilation, or diagnostics repair in general-purpose programming languages.…”

“…Last-mile repair refers to repairs that require few edits and fix syntax and simple semantic errors, such as wrong function call arity (Bavishi et al 2022). In this setting, FLAME is given the buggy formula as the input sequence, and the task is to generate the intended (valid) formula.…”

Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models

“…When querying Codex-Complete, we use three few-shot examples selected from D shot , an annotated dataset of exam-ples comprising buggy programs and desired feedback obtained by expert annotations (see Section 4.2). These annotated examples essentially provide a context to LLMCs and have shown to play an important role in optimizing the generated output (e.g., see [1,2,17,18,32]). In our case, D shot provides contextualized training data, capturing the format of how experts/tutors give explanations.…”

Fully Autonomous Programming with Large Language Models

“…However, even state-ofthe-art models fine-tuned on a specific class of programming tasks still require a costly filtering step where the LLM outputs that do not compile or pass tests are discarded [10]. These outputs tend to be superficially similar to correct solutions [11] despite failing to produce the expected output, a phenomenon known as "near miss syndrome" or "last mile problem" [12].…”