Adaptive Windows for Duplicate Detection

Draisbach, Uwe; Naumann, Felix; Szott, Sascha; Wonneberg, Oliver

doi:10.1109/icde.2012.20

Cited by 60 publications

(41 citation statements)

References 21 publications

(18 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, unnecessary comparison carried out when window size too large. To achieve effectiveness adaptive window size is used [3], [5], [9], [11]. In order to make duplication detection solution applicable, consider that adaptively plays important role.…”

Section: Sorted Neighborhood Methode and Windowingmentioning

confidence: 99%

See 1 more Smart Citation

Validation of Deduplication in Data using Similarity Measure

Wandhekar¹,

Mohanpurkar²

2015

IJCA

View full text Add to dashboard Cite

Deduplication is the process of determining all categories of information within a data set that signify the same real life / world entity. The data gathered from various resources may have data high quality issues in it. The concept to identify duplicates by using windowing and blocking strategy. The objective is to achieve better precision, good efficiency and also to reduce the false positive rate all are in accordance with the estimated similarities of records. Various Similarity metrics are commonly used to recognize the similar field entries. So the main focus of this paper is to applying appropriate similarity measure on appropriate data to properly identifying the duplicates.

show abstract

Section: Sorted Neighborhood Methode and Windowingmentioning

confidence: 99%

“…So the overall number of comparisons is getting reduced. In the past years numbers of blocking algorithms have been proposed by researchers [10], [11], [12], [13], [14]. These techniques typically form blocks or groups of observations using sorting or indexing.…”

Section: Blockingmentioning

confidence: 99%

Validation of Deduplication in Data using Similarity Measure

Wandhekar¹,

Mohanpurkar²

2015

IJCA

View full text Add to dashboard Cite

show abstract

“…Indexing techniques, discussed in more detail later, range from simple phonetic based blocking [4] and sorting of the datasets [11] to locality sensitive hashing based techniques [18,29], and unsupervised [17,26] and supervised [1,22] learning of optimal blocking schemes.…”

Section: Introductionmentioning

confidence: 99%

Using Metric Space Indexing for Complete and Efficient Record Linkage

Akgün

Dearle

Kirby

et al. 2018

Advances in Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

Abstract. Record linkage is the process of identifying records that refer to the same real-world entities in situations where entity identifiers are unavailable. Records are linked on the basis of similarity between common attributes, with every pair being classified as a link or non-link depending on their similarity. Linkage is usually performed in a threestep process: first, groups of similar candidate records are identified using indexing, then pairs within the same group are compared in more detail, and finally classified. Even state-of-the-art indexing techniques, such as locality sensitive hashing, have potential drawbacks. They may fail to group together some true matching records with high similarity, or they may group records with low similarity, leading to high computational overhead. We propose using metric space indexing (MSI) to perform complete linkage, resulting in a parameter-free process combining indexing, comparison and classification into a single step delivering complete and efficient record linkage. An evaluation on real-world data from several domains shows that linkage using MSI can yield better quality than current indexing techniques, with similar execution cost, without the need for domain knowledge or trial and error to configure the process.

show abstract

“…The recognition algorithm is composed of SNM algorithm [3], MPN algorithm [4] and KNN algorithm [5]. The SNM algorithm is a standard way to detect the similar duplicate records.…”

Section: Introductionmentioning

confidence: 99%

The SNM Algorithm Based on a Variety of Edit Distance and Variable Window

Yang¹,

Guo²,

Wang³

2017

Proceedings of the 7th International Conference on Computer Engineering and Networks — PoS(CENet2017)

View full text Add to dashboard Cite

In order to solve the drawback of SNM algorithm, which leads to inefficiency in detecting precision on approximately duplicate records when multiple sources need to be integrated, the improved SNM algorithm based on the variety of edit distance and the variable window is proposed in this paper. According to the record pattern of edit distance and the size of edit distance, the approximately duplicate records of data are deleted to reduce the number of comparisons. Then we put forward the mechanism of variable window to solve the problem. Empirical results show that this algorithm can effectively solve the problem that the window is too big or too small to leak. The experiment shows that the improved SNM algorithm can solve many problems in the integration of multi-source , and it has obvious advantages in ensuring precision and boosting efficiency.

show abstract

Adaptive Windows for Duplicate Detection

Cited by 60 publications

References 21 publications

Validation of Deduplication in Data using Similarity Measure

Validation of Deduplication in Data using Similarity Measure

Using Metric Space Indexing for Complete and Efficient Record Linkage

The SNM Algorithm Based on a Variety of Edit Distance and Variable Window

Contact Info

Product

Resources

About