Expressiveness and complexity of order dependencies

Szlichta, Jaroslaw; Godfrey, Parke; Gryz, Jarek; Zuzarte, Calisto

doi:10.14778/2556549.2556568

Cited by 29 publications

(34 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…With the help of ODs, query optimizers can eliminate costly operators such as joins and sorts, and can identify additional interesting orders: ordered streams between query operators that exploit the available indices, enable pipelining, and eliminate intermediate sorts and partitioning steps 1 http://tpc.org/tpcds/ [25]. Sorting and interesting orders are integral parts of relational query optimizers, not only for SQL order-by and group-by, but for instance also for sort-merge joins [24] and implementing indexes [6].…”

Section: Motivationmentioning

confidence: 99%

“…We assume ascending (asc) order in the lexicographical ordering, which is the SQL default for any attributes for which directionality is not explicitly indicated. We do not consider bidirectional ODs in this paper, which allow a mix of ascending and descending (desc) orders [25]. We also only consider lexicographic order specifications, as per the SQL order-by semantics, and do not consider pointwise ODs [7,8] [13,24,25] than working with pointwise ODs, because the sequence of the attributes in an order specification as in the order-by statement matters.…”

Section: Framework and Backgroundmentioning

confidence: 99%

“…Our . These types of canonical ODs have been proven to be useful to eliminate joins and simplify group-by and order-by statements [23,25].…”

Section: Fastod Main Algorithmmentioning

confidence: 99%

“…We report results over the flight dataset with 1K tuples and 40 attributes. We believe that ODs with a smaller context are more interesting generally as, for instance, they are more likely to be used in query op- timization [24,25]. Note that FASTOD discards redundant ODs if they can be inferred by ODs detected at lower levels of the lattice.…”

Section: Optimizations and Lattice Levelsmentioning

confidence: 99%

See 3 more Smart Citations

Effective and complete discovery of order dependencies via set-based axiomatization

et al. 2017

Self Cite

View full text Add to dashboard Cite

Integrity constraints (ICs) provide a valuable tool for expressing and enforcing application semantics. However, formulating constraints manually requires domain expertise, is prone to human errors, and may be excessively time consuming, especially on large datasets. Hence, proposals for automatic discovery have been made for some classes of ICs, such as functional dependencies (FDs), and recently, order dependencies (ODs). ODs properly subsume FDs, as they can additionally express business rules involving order; e.g., an employee never has a higher salary while paying lower taxes compared with another employee.We address the limitations of prior work on OD discovery which has factorial complexity in the number of attributes, is incomplete (i.e., it does not discover valid ODs that cannot be inferred from the ones found) and is not concise (i.e., it can result in "redundant" discovery and overly large discovery sets). We improve significantly on complexity, offer completeness, and define a compact canonical form. This is based on a novel polynomial mapping to a canonical form for ODs, and a sound and complete set of axioms (inference rules) for canonical ODs. This allows us to develop an efficient set-containment, lattice-driven OD discovery algorithm that uses the inference rules to prune the search space. Our algorithm has exponential worst-case time complexity, O(2 |R| ), in the number of attributes and linear complexity in the number of tuples. We prove that it produces a complete, minimal set of ODs (i.e., minimal with regards to the canonical representation). Finally, using real and synthetic datasets, we experimentally show orders-of-magnitude performance improvements over the current state-of-the-art algorithm and demonstrate effectiveness of our techniques.

show abstract

Section: Motivationmentioning

confidence: 99%

Section: Framework and Backgroundmentioning

confidence: 99%

“…Our . These types of canonical ODs have been proven to be useful to eliminate joins and simplify group-by and order-by statements [23,25].…”

Section: Fastod Main Algorithmmentioning

confidence: 99%

Section: Optimizations and Lattice Levelsmentioning

confidence: 99%

See 2 more Smart Citations

Effective and complete discovery of order dependencies via set-based axiomatization

et al. 2017

Self Cite

View full text Add to dashboard Cite

show abstract

“…al., propose a holistic data cleaning approach that allows for a unified set of data quality rules that may include FDs, Conditional Functional Dependencies (CFDs), and Matching Dependencies (MDs) [19]. This can be extended by incorporating Order Dependencies [26], [27] and Sequential Dependencies [28]. It would be interesting to extend our framework to include an expanded set of data quality rules, and to determine the necessary statistics that allow for efficient online, adaptive data cleaning.…”

Section: Discussionmentioning

confidence: 99%

Continuous data cleaning

Volkovs

Chiang²,

Szlichta³

et al. 2014

2014 IEEE 30th International Conference on Data Engineering

Self Cite

View full text Add to dashboard Cite

Abstract-In declarative data cleaning, data semantics are encoded as constraints and errors arise when the data violates the constraints. Various forms of statistical and logical inference can be used to reason about and repair inconsistencies (errors) in data. Recently, unified approaches that repair both errors in data and errors in semantics (the constraints) have been proposed. However, both data-only approaches and unified approaches are by and large static in that they apply cleaning to a single snapshot of the data and constraints. We introduce a continuous data cleaning framework that can be applied to dynamic data and constraint environments. Our approach permits both the data and its semantics to evolve and suggests repairs based on the accumulated evidence to date. Importantly, our approach uses not only the data and constraints as evidence, but also considers the past repairs chosen and applied by a user (user repair preferences). We introduce a repair classifier that predicts the type of repair needed to resolve an inconsistency, and that learns from past user repair preferences to recommend more accurate repairs in the future. Our evaluation shows that our techniques achieve high prediction accuracy and generate high quality repairs. Of independent interest, our work makes use of a set of data statistics that are shown to be sensitive to predicting particular repair types.

show abstract

Order Dependency

Szlichta¹

2018

Encyclopedia of Database Systems

View full text Add to dashboard Cite

Expressiveness and complexity of order dependencies

Cited by 29 publications

References 9 publications

Effective and complete discovery of order dependencies via set-based axiomatization

Effective and complete discovery of order dependencies via set-based axiomatization

Continuous data cleaning

Order Dependency

Contact Info

Product

Resources

About