Data profiling for data quality improvement with OpenRefine

Kusumasari, Tien Fabrianti; Fitria, Fitria

doi:10.1109/icitsi.2016.7858197

Cited by 27 publications

(8 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…OpenRefine 11 (formerly Google Refine, abbrev. OR) is a free and open-source DQ tool dedicated to data cleansing and data transformation and was discovered through (Kusumasari et al, 2016 ) in the IEEE search results, and (Tsiflidou and Manouselis, 2013 ) in the Springer Link search results as well as on GitHub 12 . While the original functionality of the tools does not primarily align with the focus of our survey, its extension MetricDoc specifically aims at assessing DQ with “customizable, reusable quality metrics in combination with immediate visual feedback” (Bors et al, 2018 ).…”

Section: Data Quality Tool Evaluationmentioning

confidence: 99%

“…While the original functionality of the tools does not primarily align with the focus of our survey, its extension MetricDoc specifically aims at assessing DQ with “customizable, reusable quality metrics in combination with immediate visual feedback” (Bors et al, 2018 ). Apart from the mention by Tsiflidou and Manouselis ( 2013 ) and Kusumasari et al ( 2016 ), OpenRefine was not evaluated in one of the previous DQ tool surveys, although it is open source. We installed the tool from GitHub and evaluated OpenRefine version 3.0 with the MetricDoc extension (where no version was provided), downloaded on February 14th, 2019.…”

Section: Data Quality Tool Evaluationmentioning

confidence: 99%

See 1 more Smart Citation

A Survey of Data Quality Measurement and Monitoring Tools

Ehrlinger

Wöß

2022

Front. Big Data

View full text Add to dashboard Cite

High-quality data is key to interpretable and trustworthy data analytics and the basis for meaningful data-driven decisions. In practical scenarios, data quality is typically associated with data preprocessing, profiling, and cleansing for subsequent tasks like data integration or data analytics. However, from a scientific perspective, a lot of research has been published about the measurement (i.e., the detection) of data quality issues and different generally applicable data quality dimensions and metrics have been discussed. In this work, we close the gap between data quality research and practical implementations with a detailed investigation on how data quality measurement and monitoring concepts are implemented in state-of-the-art tools. For the first time and in contrast to all existing data quality tool surveys, we conducted a systematic search, in which we identified 667 software tools dedicated to “data quality.” To evaluate the tools, we compiled a requirements catalog with three functionality areas: (1) data profiling, (2) data quality measurement in terms of metrics, and (3) automated data quality monitoring. Using a set of predefined exclusion criteria, we selected 13 tools (8 commercial and 5 open-source tools) that provide the investigated features and are not limited to a specific domain for detailed investigation. On the one hand, this survey allows a critical discussion of concepts that are widely accepted in research, but hardly implemented in any tool observed, for example, generally applicable data quality metrics. On the other hand, it reveals potential for functional enhancement of data quality tools and supports practitioners in the selection of appropriate tools for a given use case.

show abstract

Section: Data Quality Tool Evaluationmentioning

confidence: 99%

Section: Data Quality Tool Evaluationmentioning

confidence: 99%

A Survey of Data Quality Measurement and Monitoring Tools

Ehrlinger

Wöß

2022

Front. Big Data

View full text Add to dashboard Cite

show abstract

“…In addition, the company's receivables can continue to grow because inactive customer accounts can continue to be recorded in arrears. Thus, data cleansing is needed to overcome these problems by eliminating duplicate data, noise, inaccuracy, and data discrepancies so that companies can make proper decision making [11], [12], [13], [14].…”

Section: Introductionmentioning

confidence: 99%

Comparison of Apache SparkSQL and Oracle Performance: Case Study of Data Cleansing Process

Hidayati

Kusumasari²,

Hamami³

2022

JOIV : Int. J. Inform. Visualization

Self Cite

View full text Add to dashboard Cite

A dataset with good quality is a valuable asset for a company. The data can be processed into information to help companies improve decision-making. However, the data increased more and more over time to decrease data quality. Thus, good data management is important to keep data quality meeting company standards. One of the efforts that can be done is conducting data cleansing to clean data from errors, inaccuracies, duplication, format discrepancies, etc. Apache Spark is an engine that can analyze large amounts of data. Oracle Database is a database management system used to manage databases. Both have their own reliability and can be used to analyze SQL-shaped data. This study compared Spark and Oracle performance based on query processing time. Both were tested on queries used to perform data cleansing of millions of rows of the dataset. The research focuses on finding out Spark and Oracle's performance through quantitative analysis. The results of this study showed that there were differences in query processing times on both tools. Apache Spark is rated better because it has a relatively faster query processing time than Oracle Database. It can be concluded that Oracle is more reliable in storing complex data models than in analyzing large data. For future research, it is suggested to add other comparison aspects such as memory and CPU usage. The researchers can also consider using query optimization techniques to enrich query experiments.

show abstract

“…The result of the research shows that the Permit Number has 70 patterns on 5000 rows of data. Duplication analysis needs to be combined with other elements because one production with a single license number can be duplicated if the factory location, volume and weight of the package are different [3]. [3] There is also previous research by Febri on profiling clustering data by implementing fingerprint algorithm using BPOM dataset and tested by comparative test with result of every algorithm implemented in each application have difference.…”

Section: Introductionmentioning

confidence: 99%

“…Duplication analysis needs to be combined with other elements because one production with a single license number can be duplicated if the factory location, volume and weight of the package are different [3]. [3] There is also previous research by Febri on profiling clustering data by implementing fingerprint algorithm using BPOM dataset and tested by comparative test with result of every algorithm implemented in each application have difference. The comparative results are that Pentaho found 602 lines from 4482 lines, Talend Open Studio found 502 lines from 4482 lines and Google OpenRefine found 562 lines from 4482 lines.…”

Section: Introductionmentioning

confidence: 99%

Deduplication for Data Profiling using Open Source Platform

Gunatama¹,

Fabrianti²,

Hasibuan

2019

Proceedings of the 2018 International Conference on Industrial Enterprise and System Engineering (IcoIESE 2018)

View full text Add to dashboard Cite

Many companies still do not know of the importance of data quality for the company's improvement. Many companies in Indonesia, especially BUMN and Government companies have only single application with single database, which cause a problem related to duplication of data between columns, tables and applications when the application is integrated with other applications. This problem can be handled by doing the data preprocess, one of the data preprocess method is data profiling. Data profiling is the process of gathering information that can be determined by process or logic. The process of profiling data can be done with various tools both paid and open source tools, each has advantages both in performance and in data processing according to the desired case study. In this study, the main focus is on data analysis by conducting data profiling using deduplication method. The results of the profiling will be implemented in logical form in open source application and will do comparisons between open source applications.

show abstract

Data profiling for data quality improvement with OpenRefine

Cited by 27 publications

References 2 publications

A Survey of Data Quality Measurement and Monitoring Tools

A Survey of Data Quality Measurement and Monitoring Tools

Comparison of Apache SparkSQL and Oracle Performance: Case Study of Data Cleansing Process

Deduplication for Data Profiling using Open Source Platform

Contact Info

Product

Resources

About