FlashFill++: Scaling Programming by Example by Cutting to the Chase

Abstract

Programming-by-Examples (PBE) involves synthesizing an "intended program" from a small set of user-provided input-output examples. A key PBE strategy has been to restrict the search to a carefully designed small domain-specific language (DSL) with "effectively-invertible" (EI) operators at the top and "effectively-enumerable" (EE) operators at the bottom. This facilitates an effective combination of top-down synthesis strategy (which backpropagates outputs over various paths in the DSL using inverse functions) with a bottom-up synthesis strategy (which propagates inputs over various paths in the DSL). We address the problem of scaling synthesis to large DSLs with several non-EI/EE operators. This is motivated by the need to support a richer class of transformations and the need for readable code generation. We propose a novel solution strategy that relies on propagating fewer values and over fewer paths. Our first key idea is that of "cut functions" that prune the set of values being propagated by using knowledge of the sub-DSL on the other side. Cuts can be designed to preserve completeness of synthesis; however, DSL designers may use incomplete cuts to have finer control over the kind of programs synthesized. In either case, cuts make search feasible for non-EI/EE operators and efficient for deep DSLs. Our second key idea is that of "guarded DSLs" that allow a precedence on DSL operators, which dynamically controls exploration of various paths in the DSL. This makes search efficient over grammars with large fanouts without losing recall. It also makes ranking simpler yet more effective in learning an intended program from very few examples. Both cuts and precedence provide a mechanism to the DSL designer to restrict search to a reasonable, and possibly incomplete, space of programs. Using cuts and gDSLs, we have built FlashFill++, an industrial-strength PBE engine for performing rich string transformations, including datetime and number manipulations. The FlashFill++ gDSL is designed to enable readable code generation in different target languages including Excel's formula language, PowerFx, and Python. We show FlashFill++ is more expressive, more performant, and generates better quality code than comparable existing PBE systems. FlashFill++ is being deployed in several mass-market products ranging from spreadsheet software to notebooks and business intelligence applications, each with millions of users.

Languages with Decidable Learning: A Meta-theorem

“…This addresses a central question in expression learning with version space algebra (VSA) techniques, especially those realized as tree automata: for which symbolic languages are these learning algorithms possible? One prevailing answer in the literature is that language operators should have finite inverses (e.g., see [Cambronero et al 2023;Polozov and Gulwani 2015]). When operators have finite inverses a top-down tree automaton can be effectively constructed.…”

“…Expression learning is connected to program synthesis, and in particular, programming by example [Polozov and Gulwani 2015], where practical algorithms have been used to automate tedious programming tasks, e.g. synthesizing string programs [Cambronero et al 2023;Gulwani 2011], bit-manipulating programs from templates [Solar-Lezama et al 2006], or functional programs from examples and type information [Osera and Zdancewic 2015;Polikarpova et al 2016]. Synthesis with grammar restrictions follows work in the SyGus [Alur et al 2015], and recently, SemGuS frameworks [Kim et al 2021].…”

Experiments on Infinite Model Finding in SMT Solving

“…For example, Escher [2] is an algorithm for synthesising recursive programs by interacting with the user via input-output examples. Programmingby-example (PBE) has seen a lot of development with practical applications in spreadsheets with Flashfill and subsequent work [18,36]. Synthesis has also been applied to programming languages such as Scala [31].…”

Automated Synthesis of Decision Lists for Polynomial Specifications over Integers

“…This approach has also been applied to termination and runtime analysis of programs [15,17,16,37,10], invariant generation [18,47,25,42,30], static cost analysis of probabilistic programs [21,19,54,53,22,51], detection of deep bugs [8], and LTL model-checking [20]. A more recent but quite orthogonal paradigm is example-driven synthesis, where programs are synthesized from examples using techniques from formal methods and machine learning, e.g., [48,11,33].…”