Enabling Quality Control for Entity Resolution: A Human and Machine Cooperation Framework

Abstract

Even though many machine algorithms have been proposed for entity resolution, it remains very challenging to find a solution with quality guarantees. In this paper, we propose a novel HUman and Machine cOoperation (HUMO) framework for entity resolution (ER), which divides an ER workload between the machine and the human. HUMO enables a mechanism for quality control that can flexibly enforce both precision and recall levels. We introduce the optimization problem of HUMO, minimizing human cost given a quality requirement, and then present three optimization approaches: a conservative baseline one purely based on the monotonicity assumption of precision, a more aggressive one based on sampling and a hybrid one that can take advantage of the strengths of both previous approaches. Finally, we demonstrate by extensive experiments on real and synthetic datasets that HUMO can achieve high-quality results with reasonable return on investment (ROI) in terms of human cost, and it performs considerably better than the state-of-the-art alternatives in quality control.

r-HUMO: A Risk-Aware Human-Machine Cooperation Framework for Entity Resolution with Quality Guarantees

“…The r-HUMO framework is built on the recently proposed HUMO framework [14], [15], which can enforce quality guarantees at both precision and recall fronts. The general idea of HUMO and r-HUMO was similar to the Fellegi-Sunter theory of record linking [3], which also proposed to divide an ER workload into three parts based on match probability.…”

“…the set of instance pairs with the feature f For presentation simplicity, we summarize the frequently used notations in Table 1. Formally, we define the problem of entity resolution with quality guarantees [14], [15] as follows:…”

“…As human work is usually more expensive than machine computation, HUMO aims to minimize the workload in D H while guaranteeing resolution quality. By quantifying human cost by the number of instance pairs in D H , we define the optimization problem of HUMO as follows [14], [15]:…”

“…3 is challenging because the proportions of equivalent pairs in D + and D − are unknown, thus need to be estimated. There exist two types of approaches to minimize the size of D H : one purely based on the monotonicity assumption of precision and the other one based on sampling [14], [15]. They estimate equivalence proportion based on different assumptions.…”

“…The approach based on active learning [12], [13] can maximize recall while ensuring a pre-specified precision level. More recently, a HUman-Machine cOoperation framework [14], [15], denoted by HUMO, has been proposed to enforce more comprehensive quality guarantees at both precision and recall fronts. HUMO enables a flexible mechanism for quality control by partitioning an ER workload between the human and the machine.…”