Query Optimizer as a Service

Abstract

Query optimization is a critical technology that is common across all modern data processing systems. However, it is traditionally implemented in silos and is deeply embedded in different systems. Furthermore, over the years, query optimizers have become less understood and rarely touched pieces of code that are brittle to changes and very expensive to maintain, thus slowing down the pace of innovation. In this paper, we argue that it is time to think of query optimizer as a service in modern cloud architectures. Such a design can help build a common set of well-maintained optimizations that are externalized from the query engines and that could be learned and improved using the large workloads present in modern clouds. We present, Oasis, a reference architecture for query optimizer as a service and describe our success in deploying the early version of it in Cosmos. Finally, we discuss the risks and responsibilities involved with Oasis to ensure it is a win-win for everyone.

Towards Building Autonomous Data Services on Azure

“…ML cannot be applied without risks [19], e.g., over-indexing on a particular customer or workload, and bias is an inherent problem that we continually encounter. We introduce guardrails to protect customers from expensive solutions and from performance regressions, and we regularly check that our ML-driven decisions serve all customers fairly.…”