Spark-based Streamlined Metablocking

Araújo, Tiago Brasileiro; Pires, Carlos Eduardo Santos; Nóbrega, Thiago Pereira da

doi:10.1109/iscc.2017.8024632

Cited by 5 publications

(8 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For WNP and CNP, special care is taken to avoid redundant comparisons in the restructured blocks, while Reciprocal WNP and CNP [115] apply an aggressive pruning that retains edges satisfying the pruning criteria in both adjacent node neighborhoods. WNP and WEP are combined through the weighted sum of their thresholds in [9]. Note that the notion of Meta-blocking covers established methods that were previously considered as Block Building ones.…”

Section: Comparison Cleaningmentioning

confidence: 99%

See 1 more Smart Citation

Blocking and Filtering Techniques for Entity Resolution

et al. 2020

View full text Add to dashboard Cite

Entity Resolution (ER), a core task of Data Integration, detects different entity profiles that correspond to the same real-world object. Due to its inherently quadratic complexity, a series of techniques accelerate it so that it scales to voluminous data. In this survey, we review a large number of relevant works under two different but related frameworks: Blocking and Filtering. The former restricts comparisons to entity pairs that are more likely to match, while the latter identifies quickly entity pairs that are likely to satisfy predetermined similarity thresholds. We also elaborate on hybrid approaches that combine different characteristics. For each framework we provide a comprehensive list of the relevant works, discussing them in the greater context. We conclude with the most promising directions for future work in the field.

show abstract

Section: Comparison Cleaningmentioning

confidence: 99%

“…This strategy maximizes the efficiency of WEP and CEP, requiring just 2 and 3 jobs, respectively. Its adaptation to Apache Spark is presented in [9]. (iii) The entity-based strategy is independent of the blocking graph.…”

Section: Comparison Cleaningmentioning

confidence: 99%

Blocking and Filtering Techniques for Entity Resolution

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Araújo et al [59] have proposed a novel schema-agnostic pruning strategy called Global Weighted Node Pruning (GWPN) that combines a local threshold with a global one. The local threshold is computed for each profile as for the WNP, while the global one is computed as the average of all the edges weights.…”

Section: Related Workmentioning

confidence: 99%

“…This strategy aims to discard the edges with a low weight that connects only profiles with a very low local threshold. Compared to traditional WNP, GWNP improves precision of 0.01%, while achieving the same recall, on DBpedia dataset [59]. Araújo et al also discuss a Spark implementation for their strategy, which is based on the MapReduce parallel meta-blocking proposed in [13], and suffers from the same limitations (see Section 4.2.2).…”

Section: Related Workmentioning

confidence: 99%

Scaling entity resolution: A loosely schema-aware approach

Simonini

Gagliardelli

Bergamaschi

et al. 2019

Information Systems

View full text Add to dashboard Cite

“…For this reason, the development of efficient incremental blocking techniques able to handle streaming data appears as an open problem [13,15]. To improve efficiency and provide resources for incremental blocking techniques, parallel processing can be applied [16]. Parallel processing distributes the computational costs (i.e., to block entities) among the various resources (e.g., computers or virtual machines) of a computational infrastructure to reduce the overall execution time of blocking techniques.…”

Section: Introductionmentioning

confidence: 99%

Incremental Entity Blocking over Heterogeneous Streaming Data

et al. 2022

View full text Add to dashboard Cite

Web systems have become a valuable source of semi-structured and streaming data. In this sense, Entity Resolution (ER) has become a key solution for integrating multiple data sources or identifying similarities between data items, namely entities. To avoid the quadratic costs of the ER task and improve efficiency, blocking techniques are usually applied. Beyond the traditional challenges faced by ER and, consequently, by the blocking techniques, there are also challenges related to streaming data, incremental processing, and noisy data. To address them, we propose a schema-agnostic blocking technique capable of handling noisy and streaming data incrementally through a distributed computational infrastructure. To the best of our knowledge, there is a lack of blocking techniques that address these challenges simultaneously. This work proposes two strategies (attribute selection and top-n neighborhood entities) to minimize resource consumption and improve blocking efficiency. Moreover, this work presents a noise-tolerant algorithm, which minimizes the impact of noisy data (e.g., typos and misspellings) on blocking effectiveness. In our experimental evaluation, we use real-world pairs of data sources, including a case study that involves data from Twitter and Google News. The proposed technique achieves better results regarding effectiveness and efficiency compared to the state-of-the-art technique (metablocking). More precisely, the application of the two strategies over the proposed technique alone improves efficiency by 56%, on average.

show abstract

Spark-based Streamlined Metablocking

Cited by 5 publications

References 5 publications

Blocking and Filtering Techniques for Entity Resolution

Blocking and Filtering Techniques for Entity Resolution

Scaling entity resolution: A loosely schema-aware approach

Incremental Entity Blocking over Heterogeneous Streaming Data

Contact Info

Product

Resources

About