Scalable Matching and Clustering of Entities with FAMER

Abstract

Entity resolution identifies semantically equivalent entities, e.g. describing the same product or customer. It is especially challenging for Big Data applications where large volumes of data from many sources have to be matched and integrated. We therefore introduce a scalable entity resolution framework called FAMER (FAst Multi-source Entity Resolution system) that is based on Apache Flink for distributed execution and that can holistically match entities from multiple sources. For the latter purpose, FAMER includes multiple clustering schemes that group matching entities from different sources within clusters. In addition to previously known clustering schemes FAMER includes new approaches tailored to multi-source entity resolution. We perform a detailed comparative evaluation of eight clustering schemes for different real-life and synthetically generated datasets. The evaluation considers both the match quality as well as the scalability for different numbers of machines and data sizes.

An Overview of End-to-End Entity Resolution for Big Data

“…In the simplest case, Connected Components [80,153] is applied to compute the transitive closure of the detected matches. This naive approach increases recall, but is rather sensitive to noise.…”

“…The final task in the end-to-end ER workflow is Clustering [80,126,[153][154][155], which groups together the identified matches such that all descriptions within a cluster match. Its goal is actually to infer indirect matching relations among the detected pairs of matching descriptions so as to overcome possible limitations of the employed similarity functions.…”

Three-dimensional Entity Resolution with JedAI

“…For Dirty ER, the simplest approach is Connected Components [31,32], which sets a cut-off threshold t and considers as matches all comparisons with a similarity score higher than t; then, it estimates the transitive closure of the matches. For higher robustness to noise, more advanced algorithms build clusters around selected entities that operate as centers.…”

“…The first category includes the open-source tools that are crafted for structured data, namely Magellan [3], Dedupe [68], DuDe [69], Febrl [65], FRIL [70], OYSTER [71], Record Linkage [72] and FAMER [32]. All of them apply a budget-agnostic, schema-based end-to-end workflow that typically consists of two steps: Blocking and Matching.…”

“…LogMap [78] logic-based constraints to exclude comparisons -ISUB [79] FAMER [32] SB, SN, Q-Grams -Jaro-Winkler, TruncateBegin, TruncateEnd, EditDistance, MongeElkan, Jaccard, DICE, Overlap ExtendedJaccard, Longest Common Substring, Numerical Similarity Max Distance, Numerical Similarity Max Percentage (a) Systems for structured data KnoFuss [73] Literal Blocking -edit-distance (DATE, DiceCoefficient, Jaccard, Jaro, JaroWinkler, Overlap, MongeElkan, SmithWaterman, TokenBased, TokenWise) SERIMI [74] logic-based constraints to exclude comparisons -n-gram based [80] SB -Cosine, Jaccard (c) Systems for both structured and semi-structured data Table 6: Technical features of the main open-source ER systems. LB stands for Learning-based, LF for learning-free, C-C for Clean-Clean ER and D for Dirty ER.…”