Adaptive code generation for data-intensive analytics

Zhang, Wangda; Kim, Junyoung; Ross, Kenneth A.; Sedlar, Eric; Mössenböck, Hanspeter

doi:10.14778/3447689.3447697

Cited by 10 publications

(2 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zhang et al [90] generate efficient query execution plans for loops in user code that are marked by so-called pragmas. They use the Truffle framework to generate an AST from the loop and subsequently apply different query plans.…”

Section: Query Optimizationmentioning

confidence: 99%

Control Flow Duplication for Columnar Arrays in a Dynamic Compiler

et al. 2023

View full text Add to dashboard Cite

Columnar databases are an established way to speed up online analytical processing (OLAP) queries. Nowadays, data processing (e.g., storage, visualization, and analytics) is often performed at the programming language level, hence it is desirable to also adopt columnar data structures for common language runtimes.While there are frameworks, libraries, and APIs to enable columnar data stores in programming languages, their integration into applications typically requires developer interference. In prior work, researchers implemented an approach for automated transformation of arrays into columnar arrays in the GraalVM JavaScript runtime. However, this approach suffers from performance issues on smaller workloads as well as on more complex nested data structures. We find that the key to optimizing accesses to columnar arrays is to identify queries and apply specific optimizations to them.In this paper, we describe novel compiler optimizations in the GraalVM Compiler that optimize queries on columnar arrays. At JIT compile time, we identify loops that access potentially columnar arrays and duplicate them in order to specifically optimize accesses to columnar arrays. Additionally, we describe a new approach for creating columnar arrays from arrays consisting of complex objects by performing multi-level storage transformation. We demonstrate our approach via an implementation for JavaScript Date objects.Our work shows that automatic transformation of arrays to columnar storage is feasible even for small workloads and that more complex arrays of objects could benefit from a multi-level transformation. Furthermore, we show how we can optimize methods that handle arrays in different states by the use of duplication. We evaluated our work on microbenchmarks and established data analytics workloads (TPC-H) to demonstrate that it significantly outperforms previous efforts, with speedups of up to 10x for particular queries. Queries additionally benefit from multi-level transformation, reaching speedups of around 2x. Additionally, we show that we do not cause significant overhead on workloads not suitable for storage transformation.We argue that automatically created columnar arrays could aid developers in data-centric applications as an alternative approach to using dedicated APIs on manually created columnar arrays. Via automatic detection and optimization of queries on potentially columnar arrays, we can improve performance of data processing and further enable its use in common-particularly dynamic-programming languages. ACM CCS 2012Software and its engineering → Dynamic compilers; Runtime environments; Interpreters; Information systems → Column based storage;

show abstract

Section: Query Optimizationmentioning

confidence: 99%

Control Flow Duplication for Columnar Arrays in a Dynamic Compiler

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Techniques used in modern main-memory databases such as code generation and vectorized execution can also be used for data analytics. Zhang et al [30] present a system that automatically analyzes code to dynamically generate optimal query plans at runtime. Duta et al and Hirn et al [9,14] describe an approach that allows users to write code in the procedural programming language PL/SQL.…”

Section: Related Workmentioning

confidence: 99%

User-defined operators

Moritz

Neumann

2022

Proc. VLDB Endow.

View full text Add to dashboard Cite

In recent years, complex data mining and machine learning algorithms have become more common in data analytics. Several specialized systems exist to evaluate these algorithms on ever-growing data sets, which are built to efficiently execute different types of complex analytics queries. However, using these various systems comes at a price. Moving data out of traditional database systems is often slow as it requires exporting and importing data, which is typically performed using the relatively inefficient CSV format. Additionally, database systems usually offer strong ACID guarantees, which are lost when adding new, external systems. This disadvantage can be detrimental to the consistency of the results. Most data scientists still prefer not to use classical database systems for data analytics. The main reason why RDBMS are not used is that SQL is difficult to work with due to its declarative and set-oriented nature, and is not easily extensible. We present User-Defined Operators (UDOs) as a concept to include custom algorithms into modern query engines. Users can write idiomatic code in the programming language of their choice, which is then directly integrated into existing database systems. We show that our implementation can compete with specialized tools and existing query engines while retaining all beneficial properties of the database system.

show abstract