The Case for Learned In-Memory Joins

Abstract

In-memory join is an essential operator in any database engine. It has been extensively investigated in the database literature. In this paper, we study whether exploiting the CDF-based learned models to boost the join performance is practical or not. To the best of our knowledge, we are the first to fill this gap. We investigate the usage of CDF-based partitioning and learned indexes (e.g., RMI and RadixSpline) in the three join categories; indexed nested loop join (INLJ), sort-based joins (SJ) and hash-based joins (HJ). We proposed new efficient learned variants for the INLJ and SJ categories. In addition, we proposed a reinforcement learning based query optimizer to select the best join algorithm, whether learned or not-learned, for each join query. Our experimental analysis showed that our learned joins variants of INLJ and SJ consistently outperform the state-of-the-art techniques.

Towards Parallel Learned Sorting

“…Other works that are related to this thesis are the Learned In-Memory Joins [SK21], particularly Learned Sort-Join. Learned Sort-Join profits from the eCDF modeling approach discussed earlier to speed up both the partitioning and the chunked-join phases of the algorithm.…”

Baihe: SysML Framework for AI-driven Databases

“…In addition to the above, other lines of research have focused on learned data structures [6] used for efficiently querying large indices [10][11][12][13], automatic database configuration tuning, accurate query run time prediction and other problems [19].…”