Automatically Leveraging MapReduce Frameworks for Data-Intensive Applications

Abstract

MapReduce is a popular programming paradigm for developing large-scale, data-intensive computation. Many frameworks that implement this paradigm have recently been developed. To leverage these frameworks, however, developers must become familiar with their APIs and rewrite existing code. We present Casper, a new tool that automatically translates sequential Java programs into the MapReduce paradigm. Casper identifies potential code fragments to rewrite and translates them in two steps: (1) Casper uses program synthesis to search for a program summary (i.e., a functional specification) of each code fragment. The summary is expressed using a high-level intermediate language resembling the MapReduce paradigm and verified to be semantically equivalent to the original using a theorem prover. (2) Casper generates executable code from the summary, using either the Hadoop, Spark, or Flink API. We evaluated Casper by automatically converting realworld, sequential Java benchmarks to MapReduce. The resulting benchmarks perform up to 48.2× faster compared to the original.We implemented Casper using the Polyglot framework [37] to parse Java code into an abstract syntax tree (AST). Casper traverses the program AST to identify candidate code fragments, performs program analysis, and generates target code. We now describe the Java features supported by our compiler front-end. We also discuss how Casper identifies code fragments for translation and generates executable code from the verified program summary.

Phased synthesis of divide and conquer programs

“…To the best of our knowledge Parsynt is the only fully automatic tool that can synthesize divide-and-conquer programs of the class described in this paper from a reference implementation. A number of tools, including BIG [23], and Casper [1], synthesize various types of MapReduce [7] programs. The MapReduce model is too restrictive for splitting or partitioning divides, and all the tools mentioned fail to synthesize a solution for POP example from Section 2 or LIS example from Section 6.…”

“…We made the simplifying assumption that ⊙ = •, but in general, it is unknown. Therefore, instead of knowing that POP( ) • POP([ 1 , 2 ]) is the join expression, we have 1 Appendix C.1 spells out the rewriting steps for the interested reader. to characterize the shape of valid join expressions.…”

“…first homomorphism theorem). The literature on divide-and-conquer synthesis can be divided into two categories based on the class of input computations targeted: (1) those with list homomorphisms as input, with the aim of synthesizing efficient map-reduce [7] programs [1,14,21,23], (2) those that go beyond list homomorphisms [8,9,11,15,19,22], and target code with more dependencies. In category (2), the techniques in [8,9,11] synthesize list homomorphisms through some variation of lifting, the approach in [22] uses symbolic execution at runtime and to identify and defer dependencies, and Bellmania [15] targets input programs in the style of dynamic programming and orchestrates an efficient execution schedule to accommodate the dependencies.…”

Active Inductive Logic Programming for Code Search

“…ALICE extracts an average of 670K logic facts from each repository. The second data set is from the evaluation dataset of Casper [20], an automated code optimization technique. This dataset consists of groups of similar code fragments that follow the same data access patterns (e.g., a sequential loop over lists) and can be systematically optimized by Casper.…”

“…We evaluate ALICE using two benchmarks from prior work [9,20]. These benchmarks consist of 20 groups of similar code fragments in large-scale projects such as Eclipse JDT.…”