A Hybrid Data Deduplication Approach in Entity Resolution Using Chromatic Correlation Clustering

Applying Cluster Refinement to Improve Crowd-Based Data Duplicate Detection Approach

“…The duplicate records are put into clusters and if they don't have a common key or are noisy, then accurate deduplication becomes a challenge. Haruna et al [1], presented a hybrid based data deduplication approach in ER. Where a machine-based system, the Cosine similarity function [11], [12], was first used on sets of data to calculate for the similarity scores between each pair of records using metrics with a set threshold.…”

“…Finally, the clusters were submitted to the crowdsourcing platform, for the humans to thoroughly examine the pairs of records, to confirm their equivalence and submit their answers. Based on the crowd's confidence [9] and triangular similarity scores [1], a permanent cluster is either formed, implying the records in it are almost equal, or otherwise not formed.…”

“…However, during the cluster generation stages in the previous work [1], some of the record pairs were not issued to the crowd for examination. They were either deleted when forming the clusters or were not chosen at all for the cluster formation.…”

“…The proposed methods in this work, seek to rectify the shortcomings of work [1], by posting all record to the crowd while using additional Human Intelligent Tasks (HITs) to examine the record pairs, An HIT is where the crowd are given easy duties to execute and are given some reward. In examining all the record pairs while generating the clusters as well, there is the possibility of having a higher accuracy and efficiency during the experiments, which shall be determined and explained later in the work.…”

“…Haruna et al [1] proposed a hybrid data deduplication method, blending a machine based algorithm and a crowdsourcing platform which offered a high accuracy of deduplication, incurring lower crowd cost. Abualigah et al [17] used an algorithm called the particle swarm optimization (PSO) (FSPSOTC) to answer some feature selection problems.…”