A threshold-based similarity measure for duplicate detection

Ektefa, Mohammadreza; Jabar, Marzanah A.; Sidi, Fatimah; Memar, Sara; Ibrahim, Hamidah; Ramli, Abd Rahman

doi:10.1109/icos.2011.6079233

Cited by 15 publications

(12 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The data sets used in our study are gold standard data sets which are available from the Hasso Plattner Institute (HP) website under the Duplicate Detection Project (DuDe) [1] . These data sets have been frequently used in duplicate detection research works [40,[44][45][46][47][48][49][50][51][52][53].…”

Section: Data Setsmentioning

confidence: 99%

Missing Values Compensation in Duplicates Detection Using Hot Deck

Ali

Emran

Asmai

2021

Preprint

View full text Add to dashboard Cite

Duplicate record is a known problem within the datasets especially within databases of huge volumes. The accuracy of duplicates detection determines the efficiency of the duplicates removal process. Unfortunately, the effort to detect duplicates becomes more challenging due to the presence of missing values within the records. This is because, during the clustering and matching process, missing values can cause records that are similar to be assigned in a wrong group, causing the duplicates left undetected. In this paper, we present how duplicates detection can be improved even though missing values are present within a data set using our Duplicates Detection within the Incomplete Data set (DDID) method. We hypothetically add the missing values to the key attributes of two datasets under study, using an arbitrary pattern to simulate both complete and incomplete data sets. We analyze the results to evaluate the performance of duplicates detection using the Hot Deck method to compensate for the missing values in the key attributes. We hypothesize that by using Hot Deck, there is a performance improvement in duplicates detection. The performance of the DDID is compared with an early duplicates detection method (called DuDe) in terms of its accuracy and speed. The findings of the experiment show that, even though the data sets are incomplete, DDID is capable to offer better accuracy and faster duplicates detection as compared to a benchmark method (called DuDe). The results of this study contribute to duplicates detection under incomplete data sets constraint.

show abstract

Section: Data Setsmentioning

confidence: 99%

Missing Values Compensation in Duplicates Detection Using Hot Deck

Ali

Emran

Asmai

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Detection of Duplicate plays an important role in record linkage, near duplicate detection and filtering queue [16]. Duplication detection is used to identify the same real world entities which exist in different format or representation in database [17,18]. It is very common to find some non-identical fields or records that refer the same entity [19].…”

Section: Duplication Records Detection and Typesmentioning

confidence: 99%

An Efficient Duplication Record Detection Algorithm for Data Cleansing

Skandar¹,

Rehman²,

Anjum³

2015

IJCA

View full text Add to dashboard Cite

The purpose of this research was to review, analyze and compare algorithms lying under the empirical technique in order to suggest the most effective algorithm in terms of efficiency and accuracy. The research process was initiated by collecting the relevant research papers with the query of "duplication record detection" from IEEE database. After that, papers were categorized on the basis of different techniques proposed in the literature. In this research, the focus was made on empirical technique. The papers lying under this technique were further analyzed in order to come up with the algorithms. Finally, the comparison was performed in order to come up with the best algorithm i.e. DCS++. The selected algorithm was critically analyzed in order to improve its working. On the basis of limitations of selected algorithm, variation in algorithm was proposed and validated by developed prototype. After implementation of existing DCS++ and its proposed variation, it was found that the proposed variation in DCS++ producing better results in term of efficiency and accuracy. The algorithms lying under the empirical technique of duplicate records deduction were focused. The research material was gathered from the single digital library i.e. IEEE. A restaurant dataset was selected and the results were evaluated on the specified dataset which can be considered as a limitation of the research. The existing algorithm i.e. DCS++ and proposed variation in DCS++ were implemented in C#. As a result, it was concluded that proposed algorithm is performing outstanding than the existing algorithm.

show abstract

“…Ektefa M et al [3] have proposed a threshold-based method which takes into account both string and semantic similarity measures for comparing record pairs. The threshold-based method is experimented on a real world dataset, namely Restaurant and its effectiveness is measured based on several standard evaluation metrics.…”

Section: Review Of Related Workmentioning

confidence: 99%

“…Sooner than a threshold based method is implemented in [3], a characteristic based technique is described in the [2] for executing the deduplication in databases. Unlike from the other approaches, Elhadi M et al [4]implemented a process based on combined part of speech and improved longest common subsequence.…”

Section: Review Of Related Workmentioning

confidence: 99%

A Technique for Data Deduplication using Q-Gram Concept with Support Vector Machine

Padmanaban¹,

Bhuvaneswari²

2013

IJCA

View full text Add to dashboard Cite

Several systems that rely on consistent data to offer high quality services, such as digital libraries and e-commerce brokers, may be affected by the existence of duplicates, quasi-replicas, or near-duplicate entries in their repositories. Because of that, there have been significant investments from private and government organizations in developing methods for removing replicas from its data repositories.In this paper, we have proposed accordingly. In the previous work, duplicate record detection was done using three different similarity measures and neural network. In the previous work, we have generated feature vector based on similarity measures and then, neural network was used to find the duplicate records. In this paper, we have developed Q-gram concept with support vector machine for deduplication process. The similarity function, which we are used Dice coefficient,Damerau-Levenshtein distance,Tversky index for similarity measurement. Finally, support vector machine is used for testing whether data record is duplicate or not. A set of data generated from some similarity measures are used as the input to the proposed system. There are two processes which characterize the proposed deduplication technique, the training phase and the testing phase the experimental results showed that the proposed deduplication technique has higher accuracy than the existing method. The accuracy obtained for the proposed deduplication 88%.

show abstract

A threshold-based similarity measure for duplicate detection

Cited by 15 publications

References 17 publications

Missing Values Compensation in Duplicates Detection Using Hot Deck

Missing Values Compensation in Duplicates Detection Using Hot Deck

An Efficient Duplication Record Detection Algorithm for Data Cleansing

A Technique for Data Deduplication using Q-Gram Concept with Support Vector Machine

Contact Info

Product

Resources

About