The Communication Complexity of Distributed Set-Joins with Applications to Matrix Multiplication

Abstract

Given a set-comparison predicate P and given two lists of sets A = (A1, . . . , Am) and B = (B1, . . . , Bm), with all Ai, Bj ⊆ [n], the P-set join A P B is defined to be the set , 2, . . . , n}). When P(Ai, Bj) is the condition "Ai ∩ Bj = ∅" we call this the set-intersection-notempty join (a.k.a. the composition of A and B); when P(Ai, Bj) is "Ai ∩Bj = ∅" we call it the set-disjointness join; when P(Ai, Bj) is "Ai = Bj" we call it the set-equality join; when P(Ai, Bj) is "|Ai ∩ Bj| ≥ T " for a given threshold T , we call it the setintersection threshold join. Assuming A and B are stored at two different sites in a distributed environment, we study the (randomized) communication complexity of computing these, and related, set-joins A P B, as well as the (randomized) communication complexity of computing the exact and approximate value of their size k = |A P B|. Combined, our analyses shed new insights into the quantitative differences between these different set-joins. Furthermore, given the close affinity of the natural join and the setintersection-not-empty join, our results also yield communication complexity results for computing the natural join in a distributed environment.Additionally, we obtain new algorithms for computing the distributed set-intersection-not-empty join when the input and/or output is sparse. For instance, when the output is k-sparse, we improve anÕ(kn) communication algorithm of (Williams and Yu, SODA 2014). Observing that the set-intersection-not-empty join is isomorphic to Boolean matrix multiplication (BMM), our results imply new algorithms for fundamental graph theoretic problems related to BMM. For example, we show how to compute the transitive closure of a directed graph inÕ(k 3/2 ) time, when the transitive closure contains at most k edges. When k = O(n), we obtain a (practical)Õ(n 3/2 ) time algorithm, improving a recentÕ(n · n ω+1 4 ) time algorithm (Borassi, Crescenzi, and Habib, arXiv 2014) based on (impractical) fast matrix multiplication, where ω ≥ 2 is the exponent for matrix multiplication.

Distributed Statistical Estimation of Matrix Products with Applications

“…The studied problems include set-intersection joins, set-disjointness joins, set-equality joins, and at-least-T joins. Our results can be viewed as a significant extension to the results in [16], as well as a systematic study of classical data stream problems in the context of matrix products. In particular, [16] did not study estimating the p-norms of AB, for any p other than p = 0.…”

“…We give a 2-roundÕ(n/ϵ)-bit algorithm that approximates AB p , p ∈ [0, 2], within a (1 + ϵ) factor. For the important case of p = 0, this provides a significant improvement over the previousÕ(n/ϵ 2 ) result in [16]. Also, due to the Ω(n/ϵ 2 ) lower bound in [16] for one-round algorithms (i.e., algorithms for which Alice sends a single message to Bob, who outputs the answer), this gives a separation in the complexity of this problem for one and two-round algorithms.…”

“…In a recent paper [16] the authors and collaborators studied several join problems in the two-party communication model. The studied problems include set-intersection joins, set-disjointness joins, set-equality joins, and at-least-T joins.…”

An Optimal Algorithm for l1-Heavy Hitters in Insertion Streams and Related Problems

“…The Indexing problem is to return y ℓ for which there is a Ω(|y|) = Ω(n/ε 2 ) lower bound (Lemma 5). The initial part of the reduction follows the construction in the proof of Theorem 6 in [VWWZ15], which we encapsulate in the following lemma.…”

“…Lemma 8 (Theorem 6 in [VWWZ15]). Given y, Alice can construct a matrix P ∈ {0, 1} n×γ using public randomness, such that if P i and P j are the i'th and j'th rows of P respectively, then with probability at least 2/3, ∆(P i , P j ) Let Alice construct P according to Lemma 8 and then adjoin the bitwise complement of the matrix P below P to form the matrix P ′ ∈ {0, 1} 2n×γ ; note that each column of P ′ has exactly n 1's and n 0's.…”