No abstract

No abstract

No abstract

No abstract

Given a set-comparison predicate P and given two lists of sets A = (A1, . . . , Am) and B = (B1, . . . , Bm), with all Ai, Bj ⊆ [n], the P-set join A P B is defined to be the set , 2, . . . , n}). When P(Ai, Bj) is the condition "Ai ∩ Bj = ∅" we call this the set-intersection-notempty join (a.k.a. the composition of A and B); when P(Ai, Bj) is "Ai ∩Bj = ∅" we call it the set-disjointness join; when P(Ai, Bj) is "Ai = Bj" we call it the set-equality join; when P(Ai, Bj) is "|Ai ∩ Bj| ≥ T " for a given threshold T , we call it the setintersection threshold join. Assuming A and B are stored at two different sites in a distributed environment, we study the (randomized) communication complexity of computing these, and related, set-joins A P B, as well as the (randomized) communication complexity of computing the exact and approximate value of their size k = |A P B|. Combined, our analyses shed new insights into the quantitative differences between these different set-joins. Furthermore, given the close affinity of the natural join and the setintersection-not-empty join, our results also yield communication complexity results for computing the natural join in a distributed environment.Additionally, we obtain new algorithms for computing the distributed set-intersection-not-empty join when the input and/or output is sparse. For instance, when the output is k-sparse, we improve anÕ(kn) communication algorithm of (Williams and Yu, SODA 2014). Observing that the set-intersection-not-empty join is isomorphic to Boolean matrix multiplication (BMM), our results imply new algorithms for fundamental graph theoretic problems related to BMM. For example, we show how to compute the transitive closure of a directed graph inÕ(k 3/2 ) time, when the transitive closure contains at most k edges. When k = O(n), we obtain a (practical)Õ(n 3/2 ) time algorithm, improving a recentÕ(n · n ω+1 4 ) time algorithm (Borassi, Crescenzi, and Habib, arXiv 2014) based on (impractical) fast matrix multiplication, where ω ≥ 2 is the exponent for matrix multiplication.

We consider a number of fundamental statistical and graph problems in the message-passing model, where we have k machines (sites), each holding a piece of data, and the machines want to jointly solve a problem defined on the union of the k data sets. The communication is point-to-point, and the goal is to minimize the total communication among the k machines. This model captures all point-to-point distributed computational models with respect to minimizing communication costs. Our analysis shows that exact computation of many statistical and graph problems in this distributed setting requires a prohibitively large amount of communication, and often one cannot improve upon the communication of the simple protocol in which all machines send their data to a centralized server. Thus, in order to obtain protocols that are communication-efficient, one has to allow approximation, or investigate the distribution or layout of the data sets.

No abstract

Set reconciliation is a fundamental problem in distributed databases, where two parties each holding a set of elements wish to find their difference, so as to establish data consistency. Efficient algorithms exist for this problem with communication cost proportional only to the difference of the two sets, as opposed to the cardinality of the sets themselves. However, all existing work on set reconciliation considers two elements to be the same only if they are exactly equal. We observe that, in many applications, the elements correspond to objects on which a distance function can be defined, e.g., points in the Euclidean space, and close points often actually represent the same object. During the reconciliation, the algorithm should only find the truly different elements in the two sets while tolerating small perturbations. In this paper, we propose the robust set reconciliation problem, and take a principled approach to address this issue via the earth mover's distance. We have developed a communication and time-efficient algorithm with provable guarantees on the quality of the reconciliation. This is then complemented with an essentially matching lower bound showing the optimality of the algorithm. Our experimental results on both synthetic and real data sets have demonstrated that our algorithm also performs very well in practice.