Detecting data errors

Abstract

Data cleaning has played a critical role in ensuring data quality for enterprise applications. Naturally, there has been extensive research in this area, and many data cleaning algorithms have been translated into tools to detect and to possibly repair certain classes of errors such as outliers, duplicates, missing values, and violations of integrity constraints. Since different types of errors may coexist in the same data set, we often need to run more than one kind of tool. In this paper, we investigate two pragmatic questions: (1) are these tools robust enough to capture most errors in real-world data sets? and (2) what is the best strategy to holistically run multiple tools to optimize the detection effort? To answer these two questions, we obtained multiple data cleaning tools that utilize a variety of error detection techniques. We also collected five real-world data sets, for which we could obtain both the raw data and the ground truth on existing errors. In this paper, we report our experimental findings on the errors detected by the tools we tested. First, we show that the coverage of each tool is well below 100%. Second, we show that the order in which multiple tools are run makes a big difference. Hence, we propose a holistic multi-tool strategy that orders the invocations of the available tools to maximize their benefit, while minimizing human effort in verifying results. Third, since this holistic approach still does not lead to acceptable error coverage, we discuss two simple strategies that have the potential to improve the situation, namely domain specific tools and data enrichment. We close this paper by reasoning about the errors that are not detectable by any of the tools we tested. * Work done while doing an internship at QCRI.

Exploring and cleaning big data with random sample data blocks

“…Similarly, the proportion of erroneous categorical values can be estimated in the same way as we estimate the proportions of inconsistent values in numerical data. Furthermore, RSP-Explore can be extended directly to support logical data cleaning tasks such as those discussed in [15,54]. A block-level sample can be used to estimate the proportion of records that don't satisfy a certain constraint or the proportion of values that are slightly different from the correct value.…”

“…

Introduction

MotivationSampling-based approaches have been adopted to alleviate the burden of big data volume not only when approximate results are useful as exact ones [1][2][3][4][5], but also when the results from a small clean sample can be more accurate than those from the entire dirty data [6][7][8][9]. It is a common practice to iteratively generate small random samples of a big data set to explore the statistical properties of the entire data and define cleaning rules [10][11][12][13][14][15][16][17][18][19]. This iterative process becomes impractical or impossible on small computing clusters due to the communication, I/O and memory costs of cluster computing frameworks that implement a shared-nothing architecture [20][21][22].

…”

Data quality in customer relationship management (CRM): Literature review

“… Using the right tools has a direct impact on the performance of the adopted CRM (Alshawi et al, 2011). As different types of errors can exist in the same data set, we often need to implement more than one error detection tool (Abedjan et al, 2016). Missi et al, (2005) cite a variety of tools that can be used to achieve data quality and integration: tools that provide insight into one relational access to data, tools that transform non-relational into relational data; tools that develop, test, and perform transformations in databases and automatically generate code that makes it easy to manage even the most complex transformations of all types of data and applications; tools for converting data among hundreds of formats and applications; tools for consolidation, verification, standardization, real-time data profiling; a tool that records, models, and maintains metadata from various sources, stores numerous models and versions.…”

Artificial intelligence for ocean science data integration: current state, gaps, and way forward

“…Entity consolidation (4) is the process of merging all data about the same entity coherently (e.g., Hogan et al, 2012). An orthogonal but crucial component of the DI process is data cleansing (5), which can be applied to both the original data and the merged dataset (Abedjan et al, 2016).…”