An efficient spark-based adaptive windowing for entity matching

Mestre, Demetrio Gomes; Pires, Carlos Eduardo Santos; Nascimento, Dimas Cassimiro; Queiroz, Andreza Raquel Monteiro de; Santos, Veruska Borges; Araújo, Tiago Brasileiro

doi:10.1016/j.jss.2017.03.003

Cited by 20 publications

(12 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It works on the concept of document collections and provides a flexible storage architecture. PostGIS 12 is an open source spatial database extension for PostgreSQL Object-Relational Database Management System (ORDBMS) that adds support for geographic objects and follows the Open Geospatial Consortium's ''Simple Features for SQL Specification'' [50]. It provides several features such as processing of vector and raster data, spatial reprojection, import/export of Environmental Systems Research Institute (ESRI) shapefiles, 3D object support.…”

Section: ) Data Storage and Managementmentioning

confidence: 99%

“…It comprehensively describes the ICT solution proposed in the EUBra-BIGSEA context, the developed services and algorithms from an application-centric perspective. In this respect, it focuses on data quality and privacy aspects as well as descriptive analyt- 1 https://www.eubra-bigsea.eu/-Last visited on July 2019 ics and graphical user interface (GUI) implementation details, thus providing a novel contribution with respect to previous work ( [10], [12]- [15]).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Integrated Big and Fast Data Analytics Platform for Smart Urban Transportation Management

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

Section: ) Data Storage and Managementmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

An Integrated Big and Fast Data Analytics Platform for Smart Urban Transportation Management

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…[3], [12], and [13]. Only some initial approaches consider the use of the Apache Spark framework for distributed ER [14], [15]. FAMER utilizes Apache Flink which is similar to Apache Spark and both frameworks improve on MapReduce due to a better utilization of in-memory processing and better support for iterative algorithms as needed for clustering [16].…”

Section: Related Workmentioning

confidence: 99%

“…In the processing of source-inconsistent clusters/components we sequentially process the intra-component links (lines [12][13][14][15] in the order of their maximal link priority (determined by sortLinksByPriority in line 11) which is based on the link similarity value, link strength and link degree. The parameter conf ig in line 11 determines the weight of these three factors to compute the link priority.…”

Section: Clipmentioning

confidence: 99%

Scalable Matching and Clustering of Entities with FAMER

Saeedi¹,

Nentwig

Peukert³

et al. 2018

CSIMQ

View full text Add to dashboard Cite

Entity resolution identifies semantically equivalent entities, e.g. describing the same product or customer. It is especially challenging for Big Data applications where large volumes of data from many sources have to be matched and integrated. We therefore introduce a scalable entity resolution framework called FAMER (FAst Multi-source Entity Resolution system) that is based on Apache Flink for distributed execution and that can holistically match entities from multiple sources. For the latter purpose, FAMER includes multiple clustering schemes that group matching entities from different sources within clusters. In addition to previously known clustering schemes FAMER includes new approaches tailored to multi-source entity resolution. We perform a detailed comparative evaluation of eight clustering schemes for different real-life and synthetically generated datasets. The evaluation considers both the match quality as well as the scalability for different numbers of machines and data sizes.

show abstract

“…The SparkER tool by Gagliardelli et al [ 22 ] uses LSH, meta-blocking, and a block purging process to remove high-frequency blocking keys. Mestre et al [ 23 ] presented a sorted neighborhood implementation with an adaptive window size, which uses three Spark transformation steps to distribute the data and minimize data skew.…”

Section: Introductionmentioning

confidence: 99%

Secure Record Linkage of Large Health Data Sets: Evaluation of a Hybrid Cloud Model

Brown

Randall

2020

JMIR Med Inform

View full text Add to dashboard Cite

Background The linking of administrative data across agencies provides the capability to investigate many health and social issues with the potential to deliver significant public benefit. Despite its advantages, the use of cloud computing resources for linkage purposes is scarce, with the storage of identifiable information on cloud infrastructure assessed as high risk by data custodians. Objective This study aims to present a model for record linkage that utilizes cloud computing capabilities while assuring custodians that identifiable data sets remain secure and local. Methods A new hybrid cloud model was developed, including privacy-preserving record linkage techniques and container-based batch processing. An evaluation of this model was conducted with a prototype implementation using large synthetic data sets representative of administrative health data. Results The cloud model kept identifiers on premises and uses privacy-preserved identifiers to run all linkage computations on cloud infrastructure. Our prototype used a managed container cluster in Amazon Web Services to distribute the computation using existing linkage software. Although the cost of computation was relatively low, the use of existing software resulted in an overhead of processing of 35.7% (149/417 min execution time). Conclusions The result of our experimental evaluation shows the operational feasibility of such a model and the exciting opportunities for advancing the analysis of linkage outputs.

show abstract

An efficient spark-based adaptive windowing for entity matching

Cited by 20 publications

References 8 publications

An Integrated Big and Fast Data Analytics Platform for Smart Urban Transportation Management

An Integrated Big and Fast Data Analytics Platform for Smart Urban Transportation Management

Scalable Matching and Clustering of Entities with FAMER

Secure Record Linkage of Large Health Data Sets: Evaluation of a Hybrid Cloud Model

Contact Info

Product

Resources

About