Skew in parallel query processing

Abstract

We study the problem of computing a conjunctive query q in parallel, using p of servers, on a large database. We consider algorithms with one round of communication, and study the complexity of the communication. We are especially interested in the case where the data is skewed, which is a major challenge for scalable parallel query processing. We establish a tight connection between the fractional edge packing of the query and the amount of communication in two cases. First, in the case when the only statistics on the database are the cardinalities of the input relations, and the data is skew-free, we provide matching upper and lower bounds (up to a polylogarithmic factor of p) expressed in terms of fractional edge packings of the query q. Second, in the case when the relations are skewed and the heavy hitters and their frequencies are known, we provide upper and lower bounds expressed in terms of packings of residual queries obtained by specializing the query to a heavy hitter. All our lower bounds are expressed in the strongest form, as number of bits needed to be communicated between processors with unlimited computational power. Our results generalize prior results on uniform databases (where each relation is a matching) [4], and lower bounds for the MapReduce model [1].

Parallel-Correctness and Transferability for Conjunctive Queries

“…For these classes we conclude that it is NP-complete to decide, whether for the family F of policies associated with some given CQ Q, a CQ Q ′ is parallel-correct for all distributions from F . In Section 5.2, we will see that this, in particular, holds for the families of distribution policies related to the practical Hypercube algorithm, that was previously investigated in several works [4,6,7,10,11]. In fact, we even show that this holds for a more general class of distribution policies specified in a declarative formalism.…”

“…, k}, B i is bucket i (x i , z i ), if 6 A hash function is a partial mapping from dom to a finite set whose elements are sometimes referred to as buckets. 7 For the purpose of specification it is irrelevant whether these predicates are materialized in the database.…”

“…A factor dominating complexity in modern massively distributed database systems, however, is the number of communication steps [7]. Motivated by recent in-memory systems like Spark [1] and Shark [16], Koutris and Suciu introduced the massively parallel communication model (MPC) [14] where computation proceeds in a sequence of parallel steps each followed by global synchronization of all servers.…”

“…Recently, Beame, Koutris and Suciu [7] proved a matching upper and lower bound for the amount of communication needed to compute a full conjunctive query without self-joins in one communication round. The upper bound is provided by a randomized algorithm called Hypercube which uses a technique that can be traced back to Ganguli, Silberschatz, and Tsur [11] and is described in the context of map-reduce by Afrati and Ullman [4].…”

Optimal Broadcasting Strategies for Conjunctive Queries over Distributed Data

“…In this model, evaluation of conjunctive queries [15,7] as well as skyline queries [2] have been considered. Recently, Beame et al [8] proved a matching upper and lower bound for the amount of communication needed to compute a full conjunctive query without self-joins in one communication round. The upper bound is provided by a randomized algorithm called Hypercube which dates back to Ganguli et al [13] and was described by Afrati and Ullman [1] in the context of MapReduce algorithms.…”