Neo

Abstract

Query optimization is one of the most challenging problems in database systems. Despite the progress made over the past decades, query optimizers remain extremely complex components that require a great deal of hand-tuning for specific workloads and datasets. Motivated by this shortcoming and inspired by recent advances in applying machine learning to data management challenges, we introduce Neo ( Neural Optimizer ), a novel learning-based query optimizer that relies on deep neural networks to generate query executions plans. Neo bootstraps its query optimization model from existing optimizers and continues to learn from incoming queries, building upon its successes and learning from its failures. Furthermore, Neo naturally adapts to underlying data patterns and is robust to estimation errors. Experimental results demonstrate that Neo, even when bootstrapped from a simple optimizer like PostgreSQL, can learn a model that offers similar performance to state-of-the-art commercial optimizers, and in some cases even surpass them.

Learned garbage collection

“…Figure 3 shows the reward and memory usage behavior of each variant (omitting "Q", which performed too poorly to be graphed on the same axes as the other variants) and the CPython GC. While many reinforcement learning techniques can require hours or days to train [20,22,29], each optimized learned GC variant is able to learn a competitive policy quickly, often within seconds. The second row of Figure 3 shows memory usage over time, tracked via the CPython heap.…”

“…While these techniques can be helpful for applications where trace information is available, both works (1) require post-hoc analysis of program traces, and do not automatically adapt their policies in real time, and (2) minimize time spent in GC mechanisms, as opposed to optimizing a user-defined reward function (e.g., request latency). Many previous works have applied reinforcement learning to various systems problems, including query optimization [22,28], cluster scheduling [20], stream processing [29], software debloating [8], and cloud provisioning [21,27].…”

Qd-tree: Learning Data Layouts for Big Data Analytics

“…Further speedups may result from (1) implementing our tree library in a native language rather than in Python, and (2) switching Woodblock's learning algorithm to a distributed learner [14]. (42) p_container (34) sr_name (29) cn_name (20) p_brand (14) l_receiptdate (8) l_shipdate (5) AC 0 (2) cr_name (1) p_size (1) p_type (1) AC 2 (1) Figure 9: A Woodblock-produced top-performing qd-tree for TPC-H. The number after each legend indicates the total number of cuts on that column (or advanced cut).…”

Bounded Evaluation: Querying Big Data with Bounded Resources

“…There has been a host of work on query optimization in DBMS, including access path selection [46] , join optimization [47,48] and recently, machine leaning methods [49][50][51] . These focus on access path cost models for, e.g., main-memory concurrent systems [46] , heuristics for join [47] and group-by [48] re-ordering, learned indices [50,51] or optimizers [49,[52][53][54] . Our algorithms and techniques are complementary to the prior work, to incorporate bounded evaluation into DBMS query optimization.…”