MapReduce program synthesis

SmithCalvin,; AlbarghouthiAws,

doi:10.1145/2980983.2908102

Cited by 35 publications

(26 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are two main directions in the area of program synthesis: synthesis from informal descriptions (such as examples, natural language, or hints) (Albarghouthi et al 2013;Feng et al 2017;Feser et al 2015;Murali et al 2018;Osera and Zdancewic 2015;Polozov and Gulwani 2015;Smith and Albarghouthi 2016;Yaghmazadeh et al 2017) and synthesis from formal specifications. We will only discuss the more relevant latter direction.…”

Section: Related Workmentioning

confidence: 99%

Structuring the synthesis of heap-manipulating programs

Polikarpova

Sergey

2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

This paper describes a deductive approach to synthesizing imperative programs with pointers from declarative specifications expressed in Separation Logic. Our synthesis algorithm takes as input a pair of assertions-a preand a postcondition-which describe two states of the symbolic heap, and derives a program that transforms one state into the other, guided by the shape of the heap. The program synthesis algorithm rests on the novel framework of Synthetic Separation Logic (SSL), which generalises the classical notion of heap entailment P ⊢ Q to incorporate a possibility of transforming a heap satisfying an assertion P into a heap satisfying an assertion Q. A synthesized program represents a proof term for a transforming entailment statement P Q, and the synthesis procedure corresponds to a proof search. The derived programs are, thus, correct by construction, in the sense that they satisfy the ascribed pre/postconditions, and are accompanied by complete proof derivations, which can be checked independently.We have implemented a proof search engine for SSL in a form the program synthesizer called S SL . For efficiency, the engine exploits properties of SSL rules, such as invertibility and commutativity of rule applications on separate heaps, to prune the space of derivations it has to consider. We explain and showcase the use of SSL on characteristic examples, describe the design of S SL , and report on our experience of using it to synthesize a series of benchmark programs manipulating heap-based linked data structures. Nadia Polikarpova and Ilya Sergeythe proof search is guided by a goal in ordinary logics. In this work, we make this connection explicit and employ it for efficiently synthesizing imperative programs from SL pre-and postconditions.Structuring the Synthesis of Heap-Manipulating Programs :3 (reads as "the assertion P transforms into Q via a program c"), which unifies SL entailment P ⊢ Q and verification {P} c {Q}, with the former expressible as P Q| skip.The central practical contribution is the design and implementation of S SL -a deductive synthesizer for heap-manipulating programs, based on SSL. S SL takes as its input a library of inductive predicates, a (typically empty) list of auxiliary function specifications, and an SL specification of the function to be synthesized. It returns a-possibly recursive, but loop-free-program (in a minimalistic C-like language), which provably satisfies the given specification.Our evaluation shows that S SL can synthesize all structurally-recursive benchmarks from previous work on heap-based synthesis (Qiu and Solar-Lezama 2017), without any sketches and in most cases much faster. To the best of our knowledge, it is also the first synthesizer to automatically discover the implementations of copying linked lists and trees, and flattening a tree to a list.The essence of S SL 's synthesis algorithm is a backtracking search in the space of SSL derivations. Even though the structural constraints (i.e., the shape of the heap) embodied in the synthesis rules already pru...

show abstract

Section: Related Workmentioning

confidence: 99%

Structuring the synthesis of heap-manipulating programs

Polikarpova

Sergey

2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

show abstract

“…The problem of automatically learning programs that are consistent with a set of input-output examples has been the subject of research for the last four decades [Shaw et al 1975]. Recent advances in algorithmic and logical reasoning techniques have led to the development of PBE systems in several domains including regular expression based string transformations [Gulwani 2011;Singh 2016], data structure manipulations [Feser et al 2015;Yaghmazadeh et al 2016], network policies [Yuan et al 2014], data filtering [Wang et al 2016], file manipulations , interactive parser synthesis [Leung et al 2015], and synthesizing map-reduce distributed programs [Smith and Albarghouthi 2016]. It has also been studied from different perspectives, such as type-theoretic interpretation [Frankle et al 2016;Osera and Zdancewic 2015;Scherer and Rémy 2015], version space learning [Gulwani 2011;Polozov and Gulwani 2015], and deep learning [Devlin et al 2017;Parisotto et al 2016].…”

Section: Related Workmentioning

confidence: 99%

Program synthesis using abstraction refinement

Wang

Dillig

Singh

2017

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

We present a new approach to example-guided program synthesis based on counterexample-guided abstraction refinement. Our method uses the abstract semantics of the underlying DSL to find a program P whose abstract behavior satisfies the examples. However, since program P may be spurious with respect to the concrete semantics, our approach iteratively refines the abstraction until we either find a program that satisfies the examples or prove that no such DSL program exists. Because many programs have the same input-output behavior in terms of their abstract semantics, this synthesis methodology significantly reduces the search space compared to existing techniques that use purely concrete semantics.While synthesis using abstraction refinement (SYNGAR) could be implemented in different settings, we propose a refinement-based synthesis algorithm that uses abstract finite tree automata (AFTA). Our technique uses a coarse initial program abstraction to construct an initial AFTA, which is iteratively refined by constructing a proof of incorrectness of any spurious program. In addition to ruling out the spurious program accepted by the previous AFTA, proofs of incorrectness are also useful for ruling out many other spurious programs.We implement these ideas in a framework called Blaze, which can be instantiated in different domains by providing a suitable DSL and its corresponding concrete and abstract semantics. We have used the Blaze framework to build synthesizers for string and matrix transformations, and we compare Blaze with existing techniques. Our results for the string domain show that Blaze compares favorably with FlashFill, a domainspecific synthesizer that is now deployed in Microsoft PowerShell. In the context of matrix manipulations, we compare Blaze against Prose, a state-of-the-art general-purpose VSA-based synthesizer, and show that Blaze results in a 90x speed-up over Prose. In both application domains, Blaze also consistently improves upon the performance of two other existing techniques by at least an order of magnitude. Abstract Synthesizer Checker Refiner candidate program Failure (no solution) Synthesized program counterexample & spurious program new abstraction End-users Domain expert Examples DSL w/ abstract semantics SYNGAR It can be shown that the language of the CFTA constructed from Fig. 3 is exactly the set of abstract syntax trees (ASTs) of DSL programs that are consistent with the input-output examples. 1 Hence, once we construct such a CFTA, the synthesis task boils down to finding an AST that is accepted by the automaton. However, since there are typically many ASTs accepted by the CFTA, one can use heuristics to identify the "best" program that satisfies the input-output examples. 1 The proof can be found in the extended version of this paper [Wang et al. 2017a].AFTA constr uction AST annotated with old abstract values AST annotated with concrete values Pr oof constr uction Rank Predicates Predicates 2 AST annotated with old abstract values AST annotated with concrete values Pr oof co...

show abstract

“…The problem of automatically synthesizing programs that satisfy a given set of input-output examples has been the subject of research in the past four decades [46]. Recent advances in algorithmic and logical reasoning techniques have led to the development of PBE systems in a variety of domains including string transformations [25,47], data filtering [54], data imputation [53], data structure manipulations [23,56], matrix transformation [52], table transformations [22,27], SQL queries [51,59], and map-reduce distributed programs [48].…”

Section: Related Workmentioning

confidence: 99%

Automated migration of hierarchical data to relational tables using programming-by-example

2018

View full text Add to dashboard Cite

While many applications export data in hierarchical formats like XML and JSON, it is often necessary to convert such hierarchical documents to a relational representation. This paper presents a novel programming-by-example approach, and its implementation in a tool called Mitra, for automatically migrating tree-structured documents to relational tables. We have evaluated the proposed technique using two sets of experiments. In the first experiment, we used Mitra to automate 98 data transformation tasks collected from StackOverflow. Our method can generate the desired program for 94% of these benchmarks with an average synthesis time of 3.8 seconds. In the second experiment, we used Mitra to generate programs that can convert realworld XML and JSON datasets to full-fledged relational databases. Our evaluation shows that Mitra can automate the desired transformation for all datasets.

show abstract

MapReduce program synthesis

Cited by 35 publications

References 41 publications

Structuring the synthesis of heap-manipulating programs

Structuring the synthesis of heap-manipulating programs

Program synthesis using abstraction refinement

Automated migration of hierarchical data to relational tables using programming-by-example

Contact Info

Product

Resources

About