Generating code for holistic query evaluation

Krikellas, Konstantinos; Viglas, Stratis D.; Cintra, Marcelo

doi:10.1109/icde.2010.5447892

Cited by 108 publications

(88 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Query compilation is comparable to query plan generation in a relational DBMS, but generates and compiles highly optimized code instead of an operator plan. Previous work [12,16,19,14] has shown that query compilation results in significantly faster query processing than competing interpretative approaches (e.g., tuple-at-a-time) by providing more predictable and register/cache-friendly memory access patterns; more options for the compiler to perform loop-based optimizations; less interpretation overhead per data element; all these enhancements are present in our techniques. Query compilation is also a good fit with in-memory data processing where keeping the CPU(s) busy is one of the key challenges.…”

Section: Overviewmentioning

confidence: 71%

“…In this work we explore database-inspired strategies to make query processing on data elements in the managed memory space of a host programming language more efficient by leveraging query compilation [12,16,19]. Query compilation is a highly efficient and easily realisable way to improve the performance of languageintegrated query in managed runtimes.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Code generation for efficient query processing in managed runtimes

2014

Self Cite

View full text Add to dashboard Cite

In this paper we examine opportunities arising from the convergence of two trends in data management: in-memory database systems (IMDBs), which have received renewed attention following the availability of affordable, very large main memory systems; and language-integrated query, which transparently integrates database queries with programming languages (thus addressing the famous 'impedance mismatch' problem). Language-integrated query not only gives application developers a more convenient way to query external data sources like IMDBs, but also to use the same querying language to query an application's in-memory collections. The latter offers further transparency to developers as the query language and all data is represented in the data model of the host programming language. However, compared to IMDBs, this additional freedom comes at a higher cost for query evaluation. Our vision is to improve in-memory query processing of application objects by introducing database technologies to managed runtimes.We focus on querying and we leverage query compilation to improve query processing on application objects. We explore different query compilation strategies and study how they improve the performance of query processing over application data. We take C as the host programming language as it supports languageintegrated query through the LINQ framework. Our techniques deliver significant performance improvements over the default LINQ implementation. Our work makes important first steps towards a future where data processing applications will commonly run on machines that can store their entire datasets in-memory, and will be written in a single programming language employing languageintegrated query and IMDB-inspired runtimes to provide transparent and highly efficient querying.

show abstract

Section: Overviewmentioning

confidence: 71%

Section: Introductionmentioning

confidence: 99%

Code generation for efficient query processing in managed runtimes

2014

Self Cite

View full text Add to dashboard Cite

show abstract

“…This interpretation overhead [36,43], stemming from function calls and control flow statements that disrupt the instruction pipeline, affects pipelined query execution negatively. Intuitively, the nested relational algebra operators face similar issues.…”

Section: On-demand Query Enginesmentioning

confidence: 99%

“…Runtime code generation has become an established mechanism, used by several relational engines [6,34,36,43,45,49]. HIQUE [36] generates cache-conscious code via code templates. HyPer [43] uses the LLVM compiler [37] to generate low-level machine code.…”

Section: Related Workmentioning

confidence: 99%

Fast queries over heterogeneous data through engine customization

2016

View full text Add to dashboard Cite

Industry and academia are continuously becoming more data-driven and data-intensive, relying on the analysis of a wide variety of heterogeneous datasets to gain insights. The different data models and formats pose a significant challenge on performing analysis over a combination of diverse datasets. Serving all queries using a single, general-purpose query engine is slow. On the other hand, using a specialized engine for each heterogeneous dataset increases complexity: queries touching a combination of datasets require an integration layer over the different engines.This paper presents a system design that natively supports heterogeneous data formats and also minimizes query execution times. For multi-format support, the design uses an expressive query algebra which enables operations over various data models. For minimal execution times, it uses a code generation mechanism to mimic the system and storage most appropriate to answer a query fast. We validate our design by building Proteus, a query engine which natively supports queries over CSV, JSON, and relational binary data, and which specializes itself to each query, dataset, and workload via code generation. Proteus outperforms state-of-the-art opensource and commercial systems on both synthetic and real-world workloads without being tied to a single data model or format, all while exposing users to a single query interface.

show abstract

“…To compete with the bulk processing model in terms of CPU efficiency, HyPer relies on JiT-compilation of queries [27]. Whilst DBMS compilers have a long history [3], [8], up to recently [27], [24], [31], the focus has been flexibility and extensibility rather than performance. The idea is to generate code that is directly executable on the host system's CPU.…”

Section: B Partially Decomposed Storage In Hypermentioning

confidence: 99%

CPU and cache efficient management of memory-resident databases

Pirk

Funke²,

Grund

et al. 2013

2013 IEEE 29th International Conference on Data Engineering (ICDE)

View full text Add to dashboard Cite

Abstract-Memory-Resident Database Management Systems (MRDBMS) have to be optimized for two resources: CPU cycles and memory bandwidth. To optimize for bandwidth in mixed OLTP/OLAP scenarios, the hybrid or Partially Decomposed Storage Model (PDSM) has been proposed. However, in current implementations, bandwidth savings achieved by partial decomposition come at increased CPU costs. To achieve the aspired bandwidth savings without sacrificing CPU efficiency, we combine partially decomposed storage with Just-in-Time (JiT) compilation of queries, thus eliminating CPU inefficient function calls. Since existing cost based optimization components are not designed for JiT-compiled query execution, we also develop a novel approach to cost modeling and subsequent storage layout optimization. Our evaluation shows that the JiT-based processor maintains the bandwidth savings of previously presented hybrid query processors but outperforms them by two orders of magnitude due to increased CPU efficiency.

show abstract

Generating code for holistic query evaluation

Cited by 108 publications

References 11 publications

Code generation for efficient query processing in managed runtimes

Code generation for efficient query processing in managed runtimes

Fast queries over heterogeneous data through engine customization

CPU and cache efficient management of memory-resident databases

Contact Info

Product

Resources

About