A Data Science Framework for Data Quality Assessment and Inconsistency Detection

Abstract

The accurate analysis of data requires high-quality data. However, inconsistencies occur frequently in the actual data and lead to untrustworthy decisions in the downstream data analysis pipeline. In this research, we examine the problem of the detection of incoherence and the repair of the OMD data model (OMD). We propose a framework for data quality evaluation and an OMD repair framework. We formally define a weight-based semantile repair by deletion and have an automated weight generation system that takes into account multiple input criteria. We use multi-criteria decisions based on the correlation, contrast and conflict between multiple criteria that are often necessary in the field of data cleaning. After weight generation, we present a Min-Sum dynamic programming algorithm to find the minimum weight solution. Then we apply evolutionary optimisation techniques and use medical datasets to show improved performance that is practically feasible.

The Maturity Model of Data Quality Management in Banking Industry: PT XYZ Core System Customer Data

“…Within the framework, there are eight indicators. Based on the measurement results of customer data quality on the core system of PT XYZ, the value of each indicator is obtained; data expectation (3,8), data dimension (3,4), data policy (4,0), data procedure (3,5), data governance (3,9), data standardization (3,1), data technology (3,8), and data performance management (3,7).…”

“…The pattern formed from the integration can easily be translated and has added value for an organization or company. Data can be used in different timeframes, both short-term and can be implicated in medium-term or long-term [4][8] [9].…”