An extensible Framework for Data Cleaning

“…However, until recently few cross-references could be found between the statistical and the computer science community. While statisticians and epidemiologists speak of record or data linkage [17], the computer science and database communities often refer to the same process as data or field matching, data scrubbing, data cleaning [18,35], data cleansing [28], preprocessing, duplicate detection [5], entity uncertainty or as the object identity problem. In commercial processing of customer databases or business mailing lists, data linkage is sometimes called merge/purge processing [23], data integration [11], list washing or ETL (extraction, transformation and loading).…”

Section: Data Linkage Techniquesmentioning

confidence: 99%

“…Another possibility is to use an SQL like language [18] that allows approximate joins and cluster building of similar records, as well as decision functions that decide if two records represent the same entity. A generic knowledge-based framework based on rules and an expert system is presented in [26].…”

Section: Modern Approachesmentioning

confidence: 99%

Quality and Complexity Measures for Data Linkage and Deduplication

Christen

Goiser

2007

Studies in Computational Intelligence

151

126

View full text Add to dashboard Cite

Abstract. Deduplicating one data set or linking several data sets are increasingly important tasks in the data preparation steps of many data mining projects. The aim is to match all records relating to the same entity. Research interest in this area has increased in recent years, with techniques originating from statistics, machine learning, information retrieval, and database research being combined and applied to improve the linkage quality, as well as to increase performance and efficiency when linking or deduplicating very large data sets. Different measures have been used to characterise the quality and complexity of data linkage algorithms, and several new metrics have been proposed. An overview of the issues involved in measuring data linkage and deduplication quality and complexity is presented in this chapter. It is shown that measures in the space of record pair comparisons can produce deceptive accuracy results. Various measures are discussed and recommendations are given on how to assess data linkage and deduplication quality and complexity.

show abstract

“…[95,21,72,98,7,57,1,90,42,46,39,88]). Earlier solutions employ manually specified rules to match objects [46].…”

Section: Related Workmentioning

confidence: 99%

Mining for Information Discovery on the Web: Overview and Illustrative Research

Yu¹,

Doan²

2004

Intelligent Technologies for Information Analysis

View full text Add to dashboard Cite

Summary. The Web has become a fertile ground for numerous research activities in mining. In this chapter we discuss research on finding targeted information on the Web. First, we briefly survey the research area. We focus in particular on two key issues: (a) mining to impose structures over Web data, for example by building taxonomies and portals, to aid in Web navigation, and (b) mining to build information processing systems, such as search engines, question answering systems, and data integration ones. Next, we describe two recent Web mining projects that illustrate the use of mining techniques to address the above two key issues. We conclude by briefly discussing novel research opportunities in the area of mining for information discovery on the Web.

show abstract

An extensible Framework for Data Cleaning

Cited by 47 publications

References 12 publications

A New Architecture of Data Access Middleware under Grid Environment

A New Architecture of Data Access Middleware under Grid Environment

Quality and Complexity Measures for Data Linkage and Deduplication

Mining for Information Discovery on the Web: Overview and Illustrative Research

Contact Info

Product

Resources

About