Test-driven synthesis

Abstract

Programming-by-example technologies empower end-users to create simple programs merely by providing input/output examples. Existing systems are designed around solvers specialized for a specific set of data types or domain-specific language (DSL). We present a program synthesizer which can be parameterized by an arbitrary DSL that may contain conditionals and loops and therefore is able to synthesize programs in any domain. In order to use our synthesizer, the user provides a sequence of increasingly sophisticated input/output examples along with an expert-written DSL definition. These two inputs correspond to the two key ideas that allow our synthesizer to work in arbitrary domains. First, we developed a novel iterative synthesis technique inspired by test-driven development-which also gives our technique the name of testdriven synthesis-where the input/output examples are consumed one at a time as the program is refined. Second, the DSL allows our system to take an efficient component-based approach to enumerating possible programs. We present applications of our synthesis methodology to end-user programming for transformations over strings, XML, and table layouts. We compare our synthesizer on these applications to state-of-the-art DSL-specific synthesizers as well to the general purpose synthesizer Sketch.

Counterexample Guided Inductive Synthesis Modulo Theories

“…Another very active area of program synthesis is denoted by component-based approaches [1,[13][14][15]17,18,25]. Such approaches are concerned with assembling programs from a database of existing components and make use of various techniques, from counterexample-guided synthesis [17] to type-directed search with lightweight SMT-based deduction and partial evaluation [14] and Petri-nets [15].…”

Synthesizing program input grammars

“…Finally, our approach uses machinery related to some of the recent work on programming by example-in particular, a systematic search guided by a meta-grammar. This approach has been used to synthesize string [26], number [53], and table [27] transformations (and combinations thereof [46,47]), as well as recursive programs [1,16] and parsers [34]. Unlike these approaches, our approach exploits an oracle to reject invalid candidates.…”

Inductive programming meets the real world

“…IP developed from research on inductive program synthesis, now called inductive functional programming (IFP), and from inductive inference techniques using logic, nowadays termed inductive logic programming (ILP). IFP addresses the synthesis of recursive functional programs generalized from regularities detected in (traces of) input/output examples [42, 20] using generate-and-test approaches based on evolutionary [35,28,36] or systematic [17,29] search or data-driven analytical approaches [39,6,18,11,37,24]. Its development is complementary to efforts in synthesizing programs from complete specifications using deductive and formal methods [8].…”

“…2) or numbers. Transformation of composite data types such as tables [11] and XML [36]. (c) Formatting data [37].…”

“…Meta-synthesis frameworks can allow synthesizer writers to easily develop domain-specific synthesizers, similar to how declarative parsing frameworks like lex and yacc allow a compiler writer to easily write a parser. The FlashExtract framework [24] and the Test Driven Synthesis framework [36] allow easy development of synthesizers for extracting and transforming data from documents of various types such as text files, web pages, XML documents, tables and spreadsheets. Fig.…”