Fuzzy Joins Using MapReduce

Afrati, Foto N.; Sarma, Anish Das; Menestrina, David; Parameswaran, Aditya; Ullman, Jeffrey D.

doi:10.1109/icde.2012.66

Cited by 82 publications

(82 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…That is, the replication rate is 1, independent of the limit q on reducer size. 3 Since the replication rate is identically 1, there is no tradeoff at all between q and replication rate; i.e., the word-count problem is embarrassingly parallel, as we knew all along.…”

Section: Mapping Schemasmentioning

confidence: 99%

“…But Theorem 3.2 says that r must be at least b/ log 2 (2 b ) = 1. In [3], there is an algorithm called Splitting that, for the case of Hamming distance 1 uses 2 1+b/2 reducers, for some even b. Half of these reducers, or 2 b/2 reducers correspond to the 2 b/2 possible bit strings that may be the first half of an input string.…”

Section: Upper Bound For Hamming Distancementioning

confidence: 99%

“…A brief summary can be found in Chapter 2 of [19]. There have been many custom solutions using a single round of map-reduce for specific problems, e.g., performing fuzzy joins [3,23], clustering [7], graph analyses [2,21], multiway join [1], and so on. Here, we develop techniques for analyzing problems of this type and optimizing the performance on any distributed computing environment by explicitly studying an inherent trade-off between communication cost and parallelism.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Upper and lower bounds on the cost of a map-reduce computation

et al. 2013

Self Cite

View full text Add to dashboard Cite

In this paper we study the tradeoff between parallelism and communication cost in a map-reduce computation. For any problem that is not "embarrassingly parallel," the finer we partition the work of the reducers so that more parallelism can be extracted, the greater will be the total communication between mappers and reducers. We introduce a model of problems that can be solved in a single round of mapreduce computation. This model enables a generic recipe for discovering lower bounds on communication cost as a function of the maximum number of inputs that can be assigned to one reducer. We use the model to analyze the tradeoff for three problems: finding pairs of strings at Hamming distance d, finding triangles and other patterns in a larger graph, and matrix multiplication. For finding strings of Hamming distance 1, we have upper and lower bounds that match exactly. For triangles and many other graphs, we have upper and lower bounds that are the same to within a constant factor. For the problem of matrix multiplication, we have matching upper and lower bounds for one-round map-reduce algorithms. We are also able to explore tworound map-reduce algorithms for matrix multiplication and show that these never have more communication, for a given reducer size, than the best one-round algorithm, and often have significantly less.

show abstract

Section: Mapping Schemasmentioning

confidence: 99%

Section: Upper Bound For Hamming Distancementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Upper and lower bounds on the cost of a map-reduce computation

et al. 2013

Self Cite

View full text Add to dashboard Cite

show abstract

“…The similarity self-join problem has been studied extensively. It was introduced by Chaudhuri et al [10], and many works have followed up [4,5,7,10,27,28], including approaches that use parallel computation [1,2,6,13,24]. Discussing all these related papers is out of the scope of this work.…”

Section: Related Workmentioning

confidence: 99%

Streaming similarity self-join

Morales¹,

Gionis

2016

Proc. VLDB Endow.

View full text Add to dashboard Cite

We introduce and study the problem of computing the similarity self-join in a streaming context (sssj), where the input is an unbounded stream of items arriving continuously. The goal is to find all pairs of items in the stream whose similarity is greater than a given threshold. The simplest formulation of the problem requires unbounded memory, and thus, it is intractable. To make the problem feasible, we introduce the notion of time-dependent similarity: the similarity of two items decreases with the difference in their arrival time.By leveraging the properties of this time-dependent similarity function, we design two algorithmic frameworks to solve the sssj problem. The first one, MiniBatch (MB), uses existing index-based filtering techniques for the static version of the problem, and combines them in a pipeline. The second framework, Streaming (STR), adds time filtering to the existing indexes, and integrates new time-based bounds deeply in the working of the algorithms. We also introduce a new indexing technique (L2), which is based on an existing state-of-the-art indexing technique (L2AP), but is optimized for the streaming case.Extensive experiments show that the STR algorithm, when instantiated with the L2 index, is the most scalable option across a wide array of datasets and parameters.

show abstract

“…In [4] fuzzy joins are studied. Various algorithms that rely on a single MapReduce job are proposed, which compute all pairs of records with similarity above a certain user-specified threshold, and produce the exact result.…”

Section: Similarity Joinmentioning

confidence: 99%

A survey of large-scale analytical query processing in MapReduce

2013

View full text Add to dashboard Cite

Enterprises today acquire vast volumes of data from different sources and leverage this information by means of data analysis to support effective decision-making and provide new functionality and services. The key requirement of data analytics is scalability, simply due to the immense volume of data that need to be extracted, processed, and analyzed in a timely fashion. Arguably the most popular framework for contemporary large-scale data analytics is MapReduce, mainly due to its salient features that include scalability, fault-tolerance, ease of programming, and flexibility. However, despite its merits, MapReduce has evident performance limitations in miscellaneous analytical tasks, and this has given rise to a significant body of research that aim at improving its efficiency, while maintaining its desirable properties.This survey aims to review the state-of-the-art in improving the performance of parallel query processing using MapReduce. A set of the most significant weaknesses and limitations of MapReduce is discussed at a high level, along with solving techniques. A taxonomy is presented for categorizing existing research on MapReduce improvements according to the specific problem they target. Based on the proposed taxonomy, a clas-C. Doulkeridis

show abstract

Fuzzy Joins Using MapReduce

Cited by 82 publications

References 20 publications

Upper and lower bounds on the cost of a map-reduce computation

Upper and lower bounds on the cost of a map-reduce computation

Streaming similarity self-join

A survey of large-scale analytical query processing in MapReduce

Contact Info

Product

Resources

About