Data migration using datalog program synthesis

Wang, Yuepeng; Shah, Rushi; Criswell, Abby; Pan, Rong; Dillig, Işıl

doi:10.14778/3384345.3384350

Cited by 19 publications

(3 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Formal methods for databases. Formal methods have been used to facilitate the development and maintenance of database systems and applications, such as optimizing database applications [Cheung et al 2013;Delaware et al 2015] and data storage [Feser et al 2020], model checking database applications [Deutsch et al 2005[Deutsch et al , 2007Gligoric and Majumdar 2013], providing foundations to queries [Cheney and Ricciotti 2021;Cheney 2019, 2022], verifying correctness of schema refactoring [Wang et al 2018b], and synthesizing programs for schema evolution [Wang et al 2019[Wang et al , 2020]. Among various work, Mediator [Wang et al 2018b] is the most related to our work.…”

Section: Related Workmentioning

confidence: 99%

VeriEQL: Bounded Equivalence Verification for Complex SQL Queries with Integrity Constraints

He,

Zhao,

Wang

et al. 2024

Proc. ACM Program. Lang.

Self Cite

View full text Add to dashboard Cite

The task of SQL query equivalence checking is important in various real-world applications (including query rewriting and automated grading) that involve complex queries with integrity constraints; yet, state-of-the-art techniques are very limited in their capability of reasoning about complex features (e.g., those that involve sorting, case statement, rich integrity constraints, etc.) in real-life queries. To the best of our knowledge, we propose the first SMT-based approach and its implementation, VeriEQL, capable of proving and disproving bounded equivalence of complex SQL queries. VeriEQL is based on a new logical encoding that models query semantics over symbolic tuples using the theory of integers with uninterpreted functions. It is simple yet highly practical -- our comprehensive evaluation on over 20,000 benchmarks shows that VeriEQL outperforms all state-of-the-art techniques by more than one order of magnitude in terms of the number of benchmarks that can be proved or disproved. VeriEQL can also generate counterexamples that facilitate many downstream tasks (such as finding serious bugs in systems like MySQL and Apache Calcite).

show abstract

Section: Related Workmentioning

confidence: 99%

VeriEQL: Bounded Equivalence Verification for Complex SQL Queries with Integrity Constraints

He,

Zhao,

Wang

et al. 2024

Proc. ACM Program. Lang.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Programming by Example (PBE) is a technique that synthesizes programs from given I/O examples, and has been studied intensively in recent years [14,15,25]. PBE has been applied to help non-experts in a wide range of domains such as string manipulation [9,13], data migration [33,34], data extraction [22] and MapReduce program [28]. Specifically, our algorithm is closely related to techniques that deal with table structures as I/O examples such as SQL queries [31,36], data frame manipulation [5,11], tensor manipulation [27] and data visualization [32].…”

Section: Related Workmentioning

confidence: 99%

PATSQL: Efficient Synthesis of SQL Queries from Example Tables with Quick Inference of Projected Columns

Takenouchi,

Ishio,

Okada

et al. 2020

Preprint

View full text Add to dashboard Cite

SQL is one of the most popular tools for data analysis and used by an increasing number of users without having expertise in databases. In order to help such non-experts to write correct SQL queries, several studies have proposed programming-by-example approaches. In these approaches, the user can obtain a desired query just by giving input and output (I/O) tables as an example. While existing methods support a variety of SQL features such as aggregation and nested query, they suffer a significant increase in computational cost as the scale of I/O tables increases. In this paper, we propose an efficient algorithm that synthesizes SQL queries from I/O tables. Specifically, it has strengths in both the execution time and the scale of supported tables. We adopt a sketch-based synthesis algorithm and focus on the quick inference of the columns used in the projection operator. In particular, we restrict the structures of sketches based on transformation rules in relational algebra and propagate a novel form of constraint using the output table in a top-down manner. We implemented this algorithm in our tool PATSQL and evaluated it on 118 queries from prior benchmarks and Kaggle's tutorials. As a result, PATSQL solved 72% of the benchmarks and found 92% of the solutions within a second. Our tool is available at https://naist-se.github.io/patsql/.

show abstract

“…Seveal prior systems leveraged SMT-solvers to reason about query languages [8,19,37,48,51]; but none of these consider recursive queries. Datalog synthesizers have been described in [2,32,42,43,52]. Their setting is different from ours: the specification is given by input-output examples, and the synthesizer needs to produce a program that matches all examples.…”

Section: Introductionmentioning

confidence: 99%

Optimizing Recursive Queries with Program Synthesis

Wang¹,

Khamis²,

Ngo³

et al. 2022

Preprint

View full text Add to dashboard Cite

Most work on query optimization has concentrated on loop-free queries. However, data science and machine learning workloads today typically involve recursive or iterative computation. In this work, we propose a novel framework for optimizing recursive queries using methods from program synthesis. In particular, we introduce a simple yet powerful optimization rule called the "FGHrule" which aims to find a faster way to evaluate a recursive program. The solution is found by making use of powerful tools, such as a program synthesizer, an SMT-solver, and an equality saturation system. We demonstrate the strength of the optimization by showing that the FGH-rule can lead to speedups up to 4 orders of magnitude on three, already optimized Datalog systems.

show abstract

Data migration using datalog program synthesis

Cited by 19 publications

References 48 publications

VeriEQL: Bounded Equivalence Verification for Complex SQL Queries with Integrity Constraints

VeriEQL: Bounded Equivalence Verification for Complex SQL Queries with Integrity Constraints

PATSQL: Efficient Synthesis of SQL Queries from Example Tables with Quick Inference of Projected Columns

Optimizing Recursive Queries with Program Synthesis

Contact Info

Product

Resources

About