An Optimal Lower Bound for Distinct Elements in the Message Passing Model

Woodruff, David P.; Zhang, Qin

doi:10.1137/1.9781611973402.54

Cited by 18 publications

(18 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While we do not have a better upper bound than given by Theorem 5, we can still prove a non-trivial lower bound using the recent work [45]. We now establish a relationship between SIJ and SUM: For a fixed pair (i, j), i = j, the probability that A i ∩B j = ∅ is at most (1 − 1/16) n−1 ≤ 2 −Ω(n) (ignoring the special coordinate ).…”

Section: Lemma 5 ([19]mentioning

confidence: 87%

See 1 more Smart Citation

The Communication Complexity of Distributed Set-Joins with Applications to Matrix Multiplication

Gucht

Williams

Woodruff

et al. 2015

Proceedings of the 34th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems

Self Cite

View full text Add to dashboard Cite

Given a set-comparison predicate P and given two lists of sets A = (A1, . . . , Am) and B = (B1, . . . , Bm), with all Ai, Bj ⊆ [n], the P-set join A P B is defined to be the set , 2, . . . , n}). When P(Ai, Bj) is the condition "Ai ∩ Bj = ∅" we call this the set-intersection-notempty join (a.k.a. the composition of A and B); when P(Ai, Bj) is "Ai ∩Bj = ∅" we call it the set-disjointness join; when P(Ai, Bj) is "Ai = Bj" we call it the set-equality join; when P(Ai, Bj) is "|Ai ∩ Bj| ≥ T " for a given threshold T , we call it the setintersection threshold join. Assuming A and B are stored at two different sites in a distributed environment, we study the (randomized) communication complexity of computing these, and related, set-joins A P B, as well as the (randomized) communication complexity of computing the exact and approximate value of their size k = |A P B|. Combined, our analyses shed new insights into the quantitative differences between these different set-joins. Furthermore, given the close affinity of the natural join and the setintersection-not-empty join, our results also yield communication complexity results for computing the natural join in a distributed environment.Additionally, we obtain new algorithms for computing the distributed set-intersection-not-empty join when the input and/or output is sparse. For instance, when the output is k-sparse, we improve anÕ(kn) communication algorithm of (Williams and Yu, SODA 2014). Observing that the set-intersection-not-empty join is isomorphic to Boolean matrix multiplication (BMM), our results imply new algorithms for fundamental graph theoretic problems related to BMM. For example, we show how to compute the transitive closure of a directed graph inÕ(k 3/2 ) time, when the transitive closure contains at most k edges. When k = O(n), we obtain a (practical)Õ(n 3/2 ) time algorithm, improving a recentÕ(n · n ω+1 4 ) time algorithm (Borassi, Crescenzi, and Habib, arXiv 2014) based on (impractical) fast matrix multiplication, where ω ≥ 2 is the exponent for matrix multiplication.

show abstract

Section: Lemma 5 ([19]mentioning

confidence: 87%

“…[45]) Any randomized algorithm that computes SUM up to an additive error √ m/2 with probability δ for a sufficiently small constant δ needs Ω(mn) bits of communication.…”

mentioning

confidence: 99%

The Communication Complexity of Distributed Set-Joins with Applications to Matrix Multiplication

Gucht

Williams

Woodruff

et al. 2015

Proceedings of the 34th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…Note that under (U ,V ) ∼ ϕ, Pr[SUM(U ,V ) = 0] = Pr[SUM(U ,V ) = 1] = 1/2. Using the standard information complexity machinery (which we omit here; and can be found in for example [18,35]) we can show the following.…”

Section: A Lowermentioning

confidence: 99%

Distributed Statistical Estimation of Matrix Products with Applications

Woodruff

Zhang

2018

Proceedings of the 37th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems

Self Cite

View full text Add to dashboard Cite

We consider statistical estimations of a matrix product over the integers in a distributed setting, where we have two parties Alice and Bob; Alice holds a matrix A and Bob holds a matrix B, and they want to estimate statistics of A · B. We focus on the well-studied ℓ p -norm, distinct elements (p = 0), ℓ 0 -sampling, and heavy hitter problems. The goal is to minimize both the communication cost and the number of rounds of communication.This problem is closely related to the fundamental set-intersection join problem in databases: when p = 0 the problem corresponds to the size of the set-intersection join. When p = ∞ the output is simply the pair of sets with the maximum intersection size. When p = 1 the problem corresponds to the size of the corresponding natural join. We also consider the heavy hitters problem which corresponds to finding the pairs of sets with intersection size above a certain threshold, and the problem of sampling an intersecting pair of sets uniformly at random.

show abstract

“…Most of the work in this area has been for specific classes of G. For example, the early work of Tiwari [Tiw87] considered deterministic total communication complexity on cases of G being a path, grid or ring graph. There has been a recent surge of interest for proving lower bounds on total communication for the case when G is a star [PVZ12, WZ12, WZ13, BEO + 13, WZ14,CM15]. This work was generalized to arbitrary topology by Chattopadhyay et al [CRR14] who proved tight bounds for certain functions for all network topologies.…”

Section: Introductionmentioning

confidence: 99%

Tight Network Topology Dependent Bounds on Rounds of Communication

Chattopadhyay

Langberg

et al. 2017

Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms

View full text Add to dashboard Cite

We prove tight network topology dependent bounds on the round complexity of computing well studied k-party functions such as set disjointness and element distinctness. Unlike the usual case in the CONGEST model in distributed computing, we fix the function and then vary the underlying network topology. This complements the recent such results on total communication that have received some attention. We also present some applications to distributed graph computation problems.Our main contribution is a proof technique that allows us to reduce the problem on a general graph topology to a relevant two-party communication complexity problem. However, unlike many previous works that also used the same high level strategy, we do not reason about a twoparty communication problem that is induced by a cut in the graph. To 'stitch' back the various lower bounds from the two party communication problems, we use the notion of timed graph that has seen prior use in network coding. Our reductions use some tools from Steiner tree packing and multi-commodity flow problems that have a delay constraint.

show abstract

An Optimal Lower Bound for Distinct Elements in the Message Passing Model

Cited by 18 publications

References 19 publications

The Communication Complexity of Distributed Set-Joins with Applications to Matrix Multiplication

The Communication Complexity of Distributed Set-Joins with Applications to Matrix Multiplication

Distributed Statistical Estimation of Matrix Products with Applications

Tight Network Topology Dependent Bounds on Rounds of Communication

Contact Info

Product

Resources

About