Entity resolution on-demand

Abstract

Entity Resolution (ER) aims to identify and merge records that refer to the same real-world entity. ER is typically employed as an expensive cleaning step on the entire data before consuming it. Yet, determining which entities are useful once cleaned depends solely on the user's application, which may need only a fraction of them. For instance, when dealing with Web data, we would like to be able to filter the entities of interest gathered from multiple sources without cleaning the entire, continuously-growing data. Similarly, when querying data lakes, we want to transform data on-demand and return the results in a timely manner---a fundamental requirement of ELT ( Extract-Load-Transform ) pipelines. We propose BrewER , a framework to evaluate SQL SP queries on dirty data while progressively returning results as if they were issued on cleaned data. BrewER tries to focus the cleaning effort on one entity at a time, following an ORDER BY predicate. Thus, it inherently supports top-k and stop-and-resume execution. For a wide range of applications, a significant amount of resources can be saved. We exhaustively evaluate and show the efficacy of BrewER on four real-world datasets.

Generalized supervised meta-blocking

“…We showed that four new weighting schemes give rise to feature sets that outperform the existing ones [25], while a very small, balanced training set with just 50 labelled instances suffices for high effectiveness, high time efficiency and high scalability. In the future, we will apply our approaches to Progressive ER [29,[33][34][35].…”

BrewER: Entity Resolution On-Demand

“…The latter requirement is significantly challenging, since it demands to correctly sort the entities even before they are generated using data fusion, only relying on the original records that can produce them. In this section, we give an overview of how BrewER overcomes these challenges, while the detailed description of the algorithm is provided in the research paper [13].…”

“…We will provide users with a set of dirty datasets, composed of the reference datasets used in the research paper [13] (i.e., cameras, USB sticks, and organizations) plus several additional ones (e.g., an extended version of cameras and further datasets of commercial products from the Alaska benchmark [4] and multiple datasets from the Magellan Data Repository 2 ). These datasets cover different domains and are highly heterogeneous in terms of cleanliness, number of attributes, and number of records, ranging from the 1K records of the smallest subset of USB sticks to the 29K records of the full camera dataset, on which the batch approach would take several hours to perform the entire cleaning process [13]. Each dataset comes with its ground truth, so the users will be able to assess the efficacy (precision/recall) of each step in the ER pipeline and the correctness of the results of the given queries.…”

“…To clean only the data useful to answer the user's queries, we previously presented BrewER [13], an algorithm and a framework to perform ER in an on-demand fashion. BrewER evaluates SQL SP (Selection and Projection) queries with ordering on the dirty data and returns the results as if they were issued on the cleaned data.…”

“…Previous work had been proposed to perform ER at query time or progressively, but BrewER is the first contribution to propose a solution that combines the two [13]. In fact, query-driven approaches [1,2] aim at performing ER only on the portion of the dataset that is meaningful for answering the query, but they are not designed to support the progressive emission of the results (thus, they still operate in a batch way) and enable to use a limited range of aggregation functions (e.g., the average or the mode cannot be supported).…”