Transforming Pairwise Duplicates to Entity Clusters for High-quality Duplicate Detection

Draisbach, Uwe; Christen, Peter; Naumann, Felix

doi:10.1145/3352591

Cited by 16 publications

(14 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In many duplicate detection problems, this is a nontrivial issue, and open area of research. 6 In our case, we observe that about 98% of pairs within groups do have a similarity above the threshold.…”

Section: Methodsmentioning

confidence: 45%

More extreme duplication in FDA Adverse Event Reporting System detected by literature reference normalization and fuzzy string matching

Hung

Hauben

Essex

et al. 2022

Pharmacoepidemiology and Drug

View full text Add to dashboard Cite

Purpose Literature reports of adverse drug events can be replicated across multiple companies, resulting in extreme duplication (defined as a majority of reports being duplicates) in the FDA Adverse Event Reporting System (FAERS) database because they can escape legacy duplicate detection algorithms routinely deployed on that data source. Literature reference field, added to in 2014, could potentially be utilized to identify replicated reports. FAERS does not enforce adherence to the Vancouver referencing convention, thus the same article may be referenced differently leading to duplication. The objective of this analysis is to determine if variations of the same literature references observed in FAERS can be resolved with text normalization and fuzzy string matching. Methods We normalized the literature references recorded in the FAERS database through the first quarter of 2021 with a rule‐based algorithm so that they better conform to the Vancouver convention. Levenshtein distance was then utilized to merge sufficiently similar normalized literature references together. Results Normalization of literature references increases the percentage that can be parsed into author, title, and journal from 61.74% to 93.93%. We observe that about 98% of pairs within groups do have a Levenshtein similarity of the title above the threshold. The extreme duplication ranged from 66% to 87% with a median of 72% of reports being duplicates and often involved addictovigilance scenarios. Conclusions We have shown that these normalized references can be merged via fuzzy string matching to improve enumeration of all the individual case safety reports that refer to the same article. Inclusion of the PubMed ID and adherence to the Vancouver convention could facilitate identification of duplicates in the FAERS dataset. Awareness of this phenomenon may improve disproportionality analysis, especially in areas such as addictovigilance.

show abstract

Section: Methodsmentioning

confidence: 45%

More extreme duplication in FDA Adverse Event Reporting System detected by literature reference normalization and fuzzy string matching

Hung

Hauben

Essex

et al. 2022

Pharmacoepidemiology and Drug

View full text Add to dashboard Cite

show abstract

“…There are many ML-based solutions to data quality problems, e.g. entity resolution [25,16,1]. In this article we have shown how ML and AI bring to the table new dimensions of data quality, and how techniques from explainable AI can help in this direction.…”

Section: Final Remarksmentioning

confidence: 92%

Data Quality and Explainable AI

Bertossi¹,

Geerts

2020

J. Data and Information Quality

View full text Add to dashboard Cite

In this work, we provide some insights and develop some ideas, with few technical details, about the role of explanations in Data Quality in the context of data-based machine learning models (ML). In this direction, there are, as expected, roles for causality, and explainable artificial intelligence . The latter area not only sheds light on the models, but also on the data that support model construction. There is also room for defining, identifying, and explaining errors in data, in particular, in ML, and also for suggesting repair actions. More generally, explanations can be used as a basis for defining dirty data in the context of ML, and measuring or quantifying them. We think dirtiness as relative to the ML task at hand, e.g., classification.

show abstract

“…Other work has leveraged the graph's structure instead of just the weights between records. In [25], the authors proposed three algorithms to cluster the similarity graph based on structure rather than edge weights. They argue that graph-based transitive closure, such as in [26], produces high recall but low precision because the graph's structure is not considered during clustering.…”

Section: B Record-record Simiarlity-based Graph Entity Resolutionmentioning

confidence: 99%

ModER: Graph-based Unsupervised Entity Resolution using Composite Modularity Optimization and Locality Sensitive Hashing

Ebeid¹,

Talburt²,

Hagan³

et al. 2022

IJACSA

View full text Add to dashboard Cite

Entity resolution describes techniques used to identify documents or records that might not be duplicated; nevertheless, they might refer to the same entity. Here we study the problem of unsupervised entity resolution. Current methods rely on human input by setting multiple thresholds prior to execution. Some methods also rely on computationally expensive similarity metrics and might not be practical for big data. Hence, we focus on providing a solution, namely ModER, capable of quickly identifying entity profiles in ambiguous datasets using a graphbased approach that does not require setting a matching threshold. Our framework exploits the transitivity property of approximate string matching across multiple documents or records. We build on our previous work in graph-based unsupervised entity resolution, namely the Data Washing Machine (DWM) and the Graphbased Data Washing Machine (GDWM). We provide an extensive evaluation of a synthetic data set. We also benchmark our proposed framework using state-of-the-art methods in unsupervised entity resolution. Furthermore, we discuss the implications of the results and how it contributes to the literature.

show abstract

Transforming Pairwise Duplicates to Entity Clusters for High-quality Duplicate Detection

Cited by 16 publications

References 32 publications

More extreme duplication in FDA Adverse Event Reporting System detected by literature reference normalization and fuzzy string matching

More extreme duplication in FDA Adverse Event Reporting System detected by literature reference normalization and fuzzy string matching

Data Quality and Explainable AI

ModER: Graph-based Unsupervised Entity Resolution using Composite Modularity Optimization and Locality Sensitive Hashing

Contact Info

Product

Resources

About