MapReduce program synthesis

Smith, Calvin; Albarghouthi, Aws

doi:10.1145/2908080.2908102

Cited by 65 publications

(41 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A large body of work has been dedicated to solving program synthesis problems. Numerous systems have been developed targeting a variety of domains such as string processing [Gulwani 2011;Parisotto et al 2017], data wrangling [Feng et al 2018[Feng et al , 2017Le and Gulwani 2014], data processing [Smith and Albarghouthi 2016;Yaghmazadeh et al 2018], syntax transformations [Rolim et al 2017], database queries [Yaghmazadeh et al 2017] and bit-vector manipulations [Jha et al 2010]. We attempt to categorise these works at a coarse level according to the high-level synthesis strategy used in their respective systems.…”

Section: Related Work 71 Program Synthesismentioning

confidence: 99%

AutoPandas: neural-backed generators for program synthesis

Bavishi

Lemieux

Fox

et al. 2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

Developers nowadays have to contend with a growing number of APIs. While in the long-term they are very useful to developers, many modern APIs have an incredibly steep learning curve, due to their hundreds of functions handling many arguments, obscure documentation, and frequently changing semantics. For APIs that perform data transformations, novices can often provide an I/O example demonstrating the desired transformation, but may be stuck on how to translate it to the API. A programming-by-example synthesis engine that takes such I/O examples and directly produces programs in the target API could help such novices. Such an engine presents unique challenges due to the breadth of real-world APIs, and the often-complex constraints over function arguments. We present a generator-based synthesis approach to contend with these problems. This approach uses a program candidate generator, which encodes basic constraints on the space of programs. We introduce neural-backed operators which can be seamlessly integrated into the program generator. To improve the efficiency of the search, we simply use these operators at non-deterministic decision points, instead of relying on domain-specific heuristics. We implement this technique for the Python pandas library in AutoPandas. AutoPandas supports 119 pandas dataframe transformation functions. We evaluate AutoPandas on 26 real-world benchmarks and find it solves 17 of them. CCS Concepts: • Software and its engineering → Programming by example; Automatic programming.

show abstract

Section: Related Work 71 Program Synthesismentioning

confidence: 99%

AutoPandas: neural-backed generators for program synthesis

Bavishi

Lemieux

Fox

et al. 2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

show abstract

“…An approach closely related to ours transforms Java code to a functional IR and then to Apache Spark, after a rewriting and simplification process that, e.g., maps loop-carried dependencies to group-by operations [Radoi et al 2014]. There is also work on synthesizing MapReduce programs from sketches [Smith and Albarghouthi 2016], on defining language subsets that are guaranteed to have an efficient translation [Rompf and Brown 2017], and work in the space of just-in-time compilers to reverse-engineer Java bytecode at runtime and redirect imperative API calls to embedded DSLs .…”

Section: Related Workmentioning

confidence: 99%

Precise reasoning with structured time, structured heaps, and collective operations

Essertel

Wei

Rompf

2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

Despite decades of progress, static analysis tools still have great difficulty dealing with programs that combine arithmetic, loops, dynamic memory allocation, and linked data structures. In this paper we draw attention to two fundamental reasons for this difficulty: First, typical underlying program abstractions are low-level and inherently scalar, characterizing compound entities like data structures or results computed through iteration only indirectly. Second, to ensure termination, analyses typically project away the dimension of time, and merge information per program point, which incurs a loss in precision. As a remedy, we propose to make collective operations first-class in program analysisÐinspired by Σnotation in mathematics, and also by the success of high-level intermediate languages based on map/reduce operations in program generators and aggressive optimizing compilers for domain-specific languages (DSLs). We further propose a novel structured heap abstraction that preserves a symbolic dimension of time, reflecting the program's loop structure and thus unambiguously correlating multiple temporal points in the dynamic execution with a single point in the program text. This paper presents a formal model, based on a high-level intermediate analysis language, a practical realization in a prototype tool that analyzes C code, and an experimental evaluation that demonstrates competitive results on a series of benchmarks. Remarkably, our implementation achieves these results in a fully semantics-preserving strongest-postcondition model, which is a worst-case for analysis/verification. The underlying ideas, however, are not tied to this model and would equally apply in other settings, e.g., demand-driven invariant inference in a weakest-precondition model. Given its semantics-preserving nature, our implementation is not limited to analysis for verification, but can also check program equivalence, and translate legacy C code to high-performance DSLs. CCS Concepts: • Software and its engineering → Automated static analysis; Software verification; General programming languages; Semantics.

show abstract

“…A recent representative study on making MapReduce programming easier was a tool developed by Smith and Albarghouthi [19] for synthesizing MapReduce programs from given input-output examples on the basis of search and verification. Their approach first prepares a predefined set of data-parallel skeletons and operator templates and then extends it gradually with their compositions while verifying whether generated instances satisfy the specifications of given input-output pairs; eventually the desired programs are obtained through this iteration.…”

Section: Related Workmentioning

confidence: 99%

A Generator of Hadoop MapReduce Programs that Manipulate One-dimensional Arrays

Miyazaki

Matsuzaki

Sato

2017

Journal of Information Processing

View full text Add to dashboard Cite

MapReduce is a framework for large-scale data processing proposed by Google, and its open-source implementation, Hadoop MapReduce, is now widely used. Several language systems have been proposed to make developing MapReduce programs easier, for instance, Sawzall, FlumeJava, Pig, Hive, and Crunch. These language systems mainly target applications that can be naturally solved by using a MapReduce-like programming model. In this study, we propose a new MapReduce-program generator that accepts programs manipulating one-dimensional arrays. By using the proposed generator, users only need to write sequential programs to generate Hadoop MapReduce programs automatically. We applied some program optimization techniques to the generation of Hadoop MapReduce programs. In this paper, we also report our experiment results that compare programs generated by the proposed generator with hand-written MapReduce programs.

show abstract

MapReduce program synthesis

Cited by 65 publications

References 42 publications

AutoPandas: neural-backed generators for program synthesis

AutoPandas: neural-backed generators for program synthesis

Precise reasoning with structured time, structured heaps, and collective operations

A Generator of Hadoop MapReduce Programs that Manipulate One-dimensional Arrays

Contact Info

Product

Resources

About