Spark SQL

Abstract

Spark SQL is a new module in Apache Spark that integrates relational processing with Spark's functional programming API. Built on our experience with Shark, Spark SQL lets Spark programmers leverage the benefits of relational processing (e.g., declarative queries and optimized storage), and lets SQL users call complex analytics libraries in Spark (e.g., machine learning). Compared to previous systems, Spark SQL makes two main additions. First, it offers much tighter integration between relational and procedural processing, through a declarative DataFrame API that integrates with procedural Spark code. Second, it includes a highly extensible optimizer, Catalyst, built using features of the Scala programming language, that makes it easy to add composable rules, control code generation, and define extension points. Using Catalyst, we have built a variety of features (e.g., schema inference for JSON, machine learning types, and query federation to external databases) tailored for the complex needs of modern data analysis. We see Spark SQL as an evolution of both SQL-on-Spark and of Spark itself, offering richer APIs and optimizations while keeping the benefits of the Spark programming model.

Let High-level Graph Queries Be Parallel Efficient: An Approach Over Structural Recursion On Pregel

“…We therefore prepared two queries, named q 7 and q 8 in Spark SQL [2] using Left-Outer-Join, that query the same results as q 7 and q 8 respectively, in order to 1) validate the correctness of our parallel-efficient queries generation, 2) compare the performance of our solution to an industrial solution also under Spark implementation. Our solution is slower than Spark SQL for simple queries, e.g.…”

“…While N(2007,{}) satisfies conditions in line 2, line 5 and line 6, we thus apply line 7 and obtain: f 2 ({2007 : $gv 2 }) = if isempty($gv 2 ) then{OK : {}} else {} f 2 ({$l : $gv 2 }) = {} where entry point is f 2 .…”

How to Architect a Query Compiler

“…Compilers usually convert input programs, given as text strings, into an Intermediate Representation (IR) which contains all essential information available about the program after parsing 6 . Optimizing compilers use IRs to facilitate the definition and application of optimizations.…”

“…Recently, query compilation has returned to the limelight, with commercial systems such as StreamBase, IBM Spade, Microsoft's Hekaton, Cloudera Impala, and MemSQL employing it. Academic research has also intensified [33,2,52,56,64,53,54,55,50,84,19,62,44,6].…”

CloudMdsQL: querying heterogeneous cloud data stores with a common language

“…The operators over relational data provide a simple object-relational mapping that makes it easy to specify wrappers to the underlying RDBMS. More recently, in the context of the cloud, Spark SQL [1] has been proposed as an Apache Spark module to provide tight integration between relational and procedural processing through a declarative API that integrates relational operators with procedural Spark code, taking advantage of massive parallelism. Similarly to LINQ, Spark SQL can map to relations arbitrary Java objects as well as different data sources.…”