Robust set reconciliation

Abstract

Set reconciliation is a fundamental problem in distributed databases, where two parties each holding a set of elements wish to find their difference, so as to establish data consistency. Efficient algorithms exist for this problem with communication cost proportional only to the difference of the two sets, as opposed to the cardinality of the sets themselves. However, all existing work on set reconciliation considers two elements to be the same only if they are exactly equal. We observe that, in many applications, the elements correspond to objects on which a distance function can be defined, e.g., points in the Euclidean space, and close points often actually represent the same object. During the reconciliation, the algorithm should only find the truly different elements in the two sets while tolerating small perturbations. In this paper, we propose the robust set reconciliation problem, and take a principled approach to address this issue via the earth mover's distance. We have developed a communication and time-efficient algorithm with provable guarantees on the quality of the reconciliation. This is then complemented with an essentially matching lower bound showing the optimality of the algorithm. Our experimental results on both synthetic and real data sets have demonstrated that our algorithm also performs very well in practice.

Robust Set Reconciliation via Locality Sensitive Hashing

“…Given a communication bound of O(k log |U |) bits (where k is an input parameter), the smallest EMD(S A , S B ) one could reasonably hope to achieve is EMD(S A , S B ) = EMD k (S A , S B ). Indeed, [7] provided lower bounds for this model, which confirm that achieving EMD(S A , S B ) = EMD k (S A , S B ) requiresΩ(k log |U |) bits of communication.…”

“…We do not achieve EMD(S A , S B ) = EMD k (S A , S B ), but instead obtain a multiplicative approximation to it while usingÕ(k) communication. 2 In particular, we achieve an O(log n) approximation, improving over the O(d) approximation (where d is the dimension) of [7] for high dimensional data. (One might think the results of [7], in combination with dimension-reduction via the Johnson-Lindenstrauss lemma [16], would achieve this for, for example, the 2 norm.…”

“…Robust set reconciliation. Robust set reconciliation, introduced in [7], generalizes set reconciliation to the scenario where the set elements lie in a metric space, and sufficiently close points should be thought of as equal. As a natural example, set elements might be geometric coordinates for objects, as determined by sensors.…”

“…Other applications would include reconciling other potentially noisy data, such as databases with floating point measurements or calculations, or databases with image data that has been subjected to varying compression schemes. In such cases, the databases would not end up with the same data; but this would suffice for numerical data sets where having points that are close enough may be all that is needed [7]. This would, for example, be useful when the databases are used for machine learning via clustering or nearest neighbor search.…”

“…The first model, the Earth Mover's Distance model, was originally introduced in [7]. As in [7], we restrict ourselves to metric spaces of the form (U, f ) = ([∆] d , p ). We require that |S A | = |S B | = |S B |.…”