Avalanche-safe LINQ compilation

Abstract

We report on a query compilation technique that enables the construction of alternative efficient query providers for Microsoft's Language Integrated Query (LINQ) framework. LINQ programs are mapped into an intermediate algebraic form, suitable for execution on any SQL:1999-capable relational database system. This compilation technique leads to query providers that (1) faithfully preserve list order and nesting, both being core features of the LINQ data model, (2) support the complete family of LINQ's Standard Query Operators, (3) bring database support to LINQ to XML where the original provider performs in-memory query evaluation, and, most importantly, (4) emit SQL statement sequences whose size is only determined by the input query's result type (and thus independent of the database size).A sample query scenario uses this LINQ provider to marry database-resident TPC-H and XMark data-resulting in a unique query experience that exhibits quite promising performance characteristics, especially for large data instances. LINQ QUERY PROVIDERSLanguage Integrated Query (LINQ) seamlessly embeds a declarative query language facility into Microsoft's .NET language framework [14]. Developers use the familiar idioms of their programming language-say, C#-plus a generic set of Standard Query Operators (SQOs) to formulate queries against lists of heap-resident C# objects, XML trees, or relational tables. LINQ providers then translate these programs into an executable form chosen to match the kind of queried data: a LINQ query against database-resident relational tables is compiled into a sequence of SQL statements, for example. This translation is performed by the framework's LINQ to SQL provider. LINQ is widely perceived as a signpost to a new declarative form of data access [2,21] and the availability of efficient LINQ providers will be instrumental.The principal LINQ construct is the query comprehension, a generic iteration facility applicable to any data type Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Articles from this volume were presented at The that exhibits the properties of a monad [25] (in LINQ, this role is assumed by the generic types IQueryable<t> and IEnumerable<t>). In a single LINQ query, developers may mix and match operations against XML documents as well as relational tables and then rely on the LINQ providers in charge to perform the required data-specific operations under the hood. As of today, this conceptual appeal of the LINQ approach is not fully realized by the LINQ provider implementations that come enclosed with the .NET framework. While the current LINQ to SQL and LINQ to Entities providers emit SQL statements and thus delegat...

How to Architect a Query Compiler

“…The user base of collection programming APIs is growing. These APIs improve the integration of applications with database back-ends by making them more seamless [61,34,33]. Thus, it makes sense to consider a DSL with a collection programming API as an alternative front-end for a query compiler.…”

A step towards genuine declarative language-integrated queries

“…However, in the context of LINQ to SQL, the problem of analysing and normalising of LINQ queries in order to provide minimal and cohesive mapping to SQL has drawn attention of the scientific community. This is caused mostly by some drawbacks of the original Microsoft's solution that in some cases may fail or produce a so-called "query avalanche" [18] [19].…”

Code generation for efficient query processing in managed runtimes

“…This requires the programmer to have an understanding of query processing to write efficient LINQ queries. Query avalanches [4,9] when dealing with nested sub-queries are one of the unfortunate artifacts of this approach. Despite the lack of semantic information, some heuristic rewrites could still be applied on LINQ queries e.g., selection push-down or reordering selection predicates according to expected processing cost.…”

“…Query Q2 contains a nested sub-query. For LINQ-to-objects, we used a hand-optimized query plan that eliminates the nested sub-query to prevent LINQ-toobjects from re-evaluating it for every element and, hence, from significantly increasing the evaluation time [4,9].…”

“…Grust et al [9] performed a comprehensive analysis of the performance of LINQ queries on mixed external data sources (e.g., DBMSs and XML). They identified query avalanches as a key inefficiency of LINQ when dealing with nested queries.…”