SparkCruise

Roy, Abhishek; Jindal, Alekh; Gomatam, Priyanka; Ouyang, Xiating; Gosalia, Ashit; Ravi, Nishkam; Mann, Swinky; Jain, Pooja

doi:10.14778/3476311.3476388

Cited by 5 publications

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Deploying a Steered Query Optimizer in Production at Microsoft

Zhang

Interlandi

Mineiro

et al. 2022

Proceedings of the 2022 International Conference on Management of Data

Self Cite

View full text Add to dashboard Cite

Modern analytical workloads are highly heterogeneous and massively complex, making generic query optimizers untenable for many customers and scenarios. As a result, it is important to specialize these optimizers to instances of the workloads. In this paper, we continue a recent line of work in steering a query optimizer towards better plans for a given workload, and make major strides in pushing previous research ideas to production deployment. Along the way we solve several operational challenges including, making steering actions more manageable, keeping the costs of steering within budget, and avoiding unexpected performance regressions in production. Our resulting system, QO-Advisor, essentially externalizes the query planner to a massive offline pipeline for better exploration and specialization. We discuss various aspects of our design and show detailed results over production SCOPE workloads at Microsoft, where the system is currently enabled by default.

show abstract

Deploying a Steered Query Optimizer in Production at Microsoft

Zhang

Interlandi

Mineiro

et al. 2022

Proceedings of the 2022 International Conference on Management of Data

Self Cite

View full text Add to dashboard Cite

show abstract

Towards Building Autonomous Data Services on Azure

Zhu,

Tian,

Cahoon

et al. 2023

Companion of the 2023 International Conference on Management of Data

View full text Add to dashboard Cite

Modern cloud has turned data services into easily accessible commodities. With just a few clicks, users are now able to access a catalog of data processing systems for a wide range of tasks. However, the cloud brings in both complexity and opportunity. While cloud users can quickly start an application by using various data services, it can be difficult to configure and optimize these services to gain the most value from them. For cloud providers, managing

show abstract

Query Optimizer as a Service

Jindal¹,

Leeka²

2022

SIGMOD Rec.

View full text Add to dashboard Cite

Query optimization is a critical technology that is common across all modern data processing systems. However, it is traditionally implemented in silos and is deeply embedded in different systems. Furthermore, over the years, query optimizers have become less understood and rarely touched pieces of code that are brittle to changes and very expensive to maintain, thus slowing down the pace of innovation. In this paper, we argue that it is time to think of query optimizer as a service in modern cloud architectures. Such a design can help build a common set of well-maintained optimizations that are externalized from the query engines and that could be learned and improved using the large workloads present in modern clouds. We present, Oasis, a reference architecture for query optimizer as a service and describe our success in deploying the early version of it in Cosmos. Finally, we discuss the risks and responsibilities involved with Oasis to ensure it is a win-win for everyone.

show abstract

SparkCruise

Cited by 5 publications

References 16 publications

Deploying a Steered Query Optimizer in Production at Microsoft

Deploying a Steered Query Optimizer in Production at Microsoft

Towards Building Autonomous Data Services on Azure

Query Optimizer as a Service

Contact Info

Product

Resources

About