Diverse near neighbor problem

Abbar, Sofiane; Amer-Yahia, Sihem; Indyk, Piotr; Mahabadi, Sepideh; Varadarajan, Kasturi

doi:10.1145/2493132.2462401

Cited by 12 publications

(57 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence, the vertex v only needs to broadcast this hash function to its adjacent edges which requires polylog(n) bits for representation (see, e.g. [64]) and thus can be done in O(1) MPC rounds on machines of memory n Ω (1) . We then send all edges assigned to one subgraph to a dedicated machine.…”

Section: Mpc Implementation Of the Parallel Algorithmmentioning

confidence: 99%

See 1 more Smart Citation

Coresets Meet EDCS: Algorithms for Matching and Vertex Cover on Massive Graphs

Assadi¹,

Bateni²,

Bernstein³

et al. 2019

Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms

View full text Add to dashboard Cite

As massive graphs become more prevalent, there is a rapidly growing need for scalable algorithms that solve classical graph problems, such as maximum matching and minimum vertex cover, on large datasets. For massive inputs, several different computational models have been introduced, including the streaming model, the distributed communication model, and the massively parallel computation (MPC) model that is a common abstraction of MapReduce-style computation. In each model, algorithms are analyzed in terms of resources such as space used or rounds of communication needed, in addition to the more traditional approximation ratio.In this paper, we give a single unified approach that yields better approximation algorithms for matching and vertex cover in all these models. The highlights include:• The first one pass, significantly-better-than-2-approximation for matching in random arrival streams that uses subquadratic space, namely a (1.5 + ε)-approximation streaming algorithm that uses O(n 1.5 ) space for constant ε > 0. • The first 2-round, better-than-2-approximation for matching in the MPC model that uses subquadratic space per machine, namely a (1.5 + ε)-approximation algorithm with O( √ mn + n) memory per machine for constant ε > 0.By building on our unified approach, we further develop parallel algorithms in the MPC model that give a (1 + ǫ)-approximation to matching and an O(1)-approximation to vertex cover in only O(log log n) MPC rounds and O(n/polylog(n)) memory per machine. These results settle multiple open questions posed in the recent paper of Czumaj et al. [STOC 2018].We obtain our results by a novel combination of two previously disjoint set of techniques, namely randomized composable coresets and edge degree constrained subgraphs (EDCS). We significantly extend the power of these techniques and prove several new structural results. For example, we show that an EDCS is a sparse certificate for large matchings and small vertex covers that is quite robust to sampling and composition. * sassadi@cis.upenn.edu.As massive graphs become more prevalent, there is a rapidly growing need for scalable algorithms that solve classical graph problems on large datasets. When dealing with massive data, the entire input graph is orders of magnitude larger than the amount of storage on one processor and hence any algorithm needs to explicitly address this issue. For massive inputs, several different computational models have been introduced, each focusing on certain additional resources needed to solve largescale problems. Some examples include the streaming model, the distributed communication model, and the massively parallel computation (MPC) model that is a common abstraction of MapReducestyle computation (see Section 2 for a definition of MPC). The target resources in these models are the number of rounds of communication and the local storage on each machine.Given the variety of relevant models, there has been a lot of attention on designing general algorithmic techniques that can be applicable across a wide ran...

show abstract

Section: Mpc Implementation Of the Parallel Algorithmmentioning

confidence: 99%

“…We now define randomized composable coresets in more detail; for brevity, we refer to them as randomized coresets. Given a graph G(V, E), with m = |E| and n = |V |, consider a random partition of E into k edge sets E (1) , . .…”

mentioning

confidence: 99%

Coresets Meet EDCS: Algorithms for Matching and Vertex Cover on Massive Graphs

Assadi¹,

Bateni²,

Bernstein³

et al. 2019

Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms

View full text Add to dashboard Cite

show abstract

“…However, two prominent problems are notably absent from the list of successes, namely, the maximum matching problem and the minimum vertex cover problem. Indeed, it was shown recently [10] that both matching and vertex cover require summaries of size n 2−o (1) for even computing a polylog(n)-approximate solution 2 .…”

Section: Introductionmentioning

confidence: 99%

Randomized Composable Coresets for Matching and Vertex Cover

Assadi

Khanna

2017

Proceedings of the 29th ACM Symposium on Parallelism in Algorithms and Architectures

View full text Add to dashboard Cite

A common approach for designing scalable algorithms for massive data sets is to distribute the computation across, say k, machines and process the data using limited communication between them. A particularly appealing framework here is the simultaneous communication model whereby each machine constructs a small representative summary of its own data and one obtains an approximate/exact solution from the union of the representative summaries. If the representative summaries needed for a problem are small, then this results in a communication-efficient and round-optimal (requiring essentially no interaction between the machines) protocol. Some well-known examples of techniques for creating summaries include sampling, linear sketching, and composable coresets. These techniques have been successfully used to design communication efficient solutions for many fundamental graph problems. However, two prominent problems are notably absent from the list of successes, namely, the maximum matching problem and the minimum vertex cover problem. Indeed, it was shown recently that for both these problems, even achieving a modest approximation factor of polylog(n) requires using representative summaries of size Ω(n 2 ) i.e. essentially no better summary exists than each machine simply sending its entire input graph.The main insight of our work is that the intractability of matching and vertex cover in the simultaneous communication model is inherently connected to an adversarial partitioning of the underlying graph across machines. We show that when the underlying graph is randomly partitioned across machines, both these problems admit randomized composable coresets of size O(n) that yield an O(1)-approximate solution 1 . In other words, a small subgraph of the input graph at each machine can be identified as its representative summary and the final answer then is obtained by simply running any maximum matching or minimum vertex cover algorithm on these combined subgraphs. This results in an O(1)-approximation simultaneous protocol for these problems with O(nk) total communication when the input is randomly partitioned across k machines. We also prove our results are optimal in a very strong sense: we not only rule out existence of smaller randomized composable coresets for these problems but in fact show that our O(nk) bound for total communication is optimal for any simultaneous communication protocol (i.e. not only for randomized coresets) for these two problems. Finally, by a standard application of composable coresets, our results also imply MapReduce algorithms with the same approximation guarantee in one or two rounds of communication, improving the previous best known round complexity for these problems.

show abstract

“…Diversity Indices We measure sample diversity using the Shannon index (1), inverse Simpson index (2), and Berger-Parker index (3).…”

Section: Analysis Of the Diversity Indexmentioning

confidence: 99%

“…Our aim is to perform diverse sampling while achieving the above three properties. Diversity maximization problems have been explored by the computational geometry community in the context of diverse coresets [11] and diverse near-neighbor search [2]. Diversity is usually defined geometrically.…”

Section: Introductionmentioning

confidence: 99%

Diversified RACE Sampling on Data Streams Applied to Metagenomic Sequence Analysis

Coleman

Geordie

Chou

et al. 2019

Preprint

View full text Add to dashboard Cite

The rise of whole-genome shotgun sequencing (WGS) has enabled numerous breakthroughs in large-scale comparative genomics research. However, the size of genomic datasets has grown exponentially over the last few years, leading to new challenges for traditional streaming algorithms. Modern petabyte-sized genomic datasets are difficult to process because they are delivered by high-throughput data streams and are difficult to store. As a result, many traditional streaming problems are becoming increasingly relevant. One such problem is the task of constructing a maximally diverse sample over a data stream. In this regime, complex sampling procedures are not possible due to the overwhelming data generation rate. In theory, the best diversity sampling methods are based on a simple greedy algorithm that compares the current sequence with a large pool of sampled sequences and decides whether to accept or reject the sequence. While these methods are elegant and optimal, they are largely confined to the theoretical realm because the greedy procedure is too slow in practice. While there are many methods to identify common elements in data streams efficiently, fast and memory-efficient diversity sampling remains a challenging and fundamental data streaming problem with few satisfactory solutions. In this work, we bridge the gap with RACE sampling, an online algorithm for diversified sampling. Unlike random sampling, which samples uniformly, RACE selectively accepts samples from streams that lead to higher sequence diversity. At the same time, RACE is as computationally efficient as random sampling and avoids pairwise similarity comparisons between sequences. At the heart of RACE lies an efficient lookup array constructed using locality-sensitive hashing (LSH). Our theory indicates that an accept/reject procedure based on LSH lookups is sufficient to obtain a highly diverse subsample. We provide rigorous theoretical guarantees for well-known biodiversity indices and show that RACE can nearly double the Shannon and Simpson indices of a genetic sample in practice, all while using the same resources as random sampling. We also compare RACE against Diginorm and coreset-based diversity sampling methods and find that RACE is faster and more memory efficient. Our algorithm is straightforward to implement, easy to parallelize, and fast enough to keep pace with the overwhelming data generation rates. We expect that as DNA sequence data streams become more mainstream and faster, RACE will become an essential component for many applications. 1

show abstract

Diverse near neighbor problem

Cited by 12 publications

References 8 publications

Coresets Meet EDCS: Algorithms for Matching and Vertex Cover on Massive Graphs

Coresets Meet EDCS: Algorithms for Matching and Vertex Cover on Massive Graphs

Randomized Composable Coresets for Matching and Vertex Cover

Diversified RACE Sampling on Data Streams Applied to Metagenomic Sequence Analysis

Contact Info

Product

Resources

About