Making queries tractable on big data with preprocessing

Abstract

A query class is traditionally considered tractable if there exists a polynomial-time (PTIME) algorithm to answer its queries. When it comes to big data, however, PTIME algorithms often become infeasible in practice. A traditional and effective approach to coping with this is to preprocess data off-line, so that queries in the class can be subsequently evaluated on the data efficiently. This paper aims to provide a formal foundation for this approach in terms of computational complexity. (1) We propose a set of Π-tractable queries, denoted by ΠT 0 Q , to characterize classes of queries that can be answered in parallel poly-logarithmic time (NC) after PTIME preprocessing. (2) We show that several natural query classes are Π-tractable and are feasible on big data. (3) We also study a set ΠT Q of query classes that can be effectively converted to Π-tractable queries by re-factorizing its data and queries for preprocessing. We introduce a form of NC reductions to characterize such conversions. (4) We show that a natural query class is complete for ΠT Q . (5) We also show that ΠT 0 Q ⊂ P unless P = NC, i.e., the set ΠT 0 Q of all Π-tractable queries is properly contained in the set P of all PTIME queries. Nonetheless, ΠT Q = P, i.e., all PTIME query classes can be made Π-tractable via proper refactorizations. This work is a step towards understanding the tractability of queries in the context of big data.

On scale independence for querying big data

“…For instance, when Q is in CQ and constraints in Σ are expressed in CQ, RCDP is NEXPTIME-complete, while QDSI is Σ p 3 -complete. There has also been recent work on querying big data, e.g., on the communication complexity of parallel query evaluation [17,18], the complexity of query processing in terms of MapReduce rounds [2,30], and the study of query classes that are tractable on big data [13]. In contrast, this work studies whether it is feasible to compute query answers in big data by accessing a small subset of the data, and if so, how to efficiently identify this subset.…”

When Good-Enough is Enough: Complex Queries at Fixed Cost

“…al. [9], [10] formally defines the concept of query practicality through their definition of Π-tractability. A query Q is Π-tractable if there exists a polynomial-time algorithm to transform a data set D into D such that Q(D ) can be computed in polylog-time (O((log n) k ) for some k).…”

Querying Big Data: Bridging Theory and Practice

“…To this end, we propose a notion of BD-tractable queries [8] , to help us determine what queries are tractable or feasible on big data.…”

“…The revisions are defined in terms of computational costs [8] , communication (coordination) rounds [34][35] , or MapReduce steps [31] and data shipments [36] in the MapReduce framework [37] . Our notions of BD-tractability focus on computational costs [8] . The study is still preliminary, and a number of questions remain open.…”