Towards a Data Quality Framework for Heterogeneous Data

Micic, Natasha; Neagu, Daniel; Campean, Felician; Zadeh, Esmaeil Habib

doi:10.1109/ithings-greencom-cpscom-smartdata.2017.28

Cited by 19 publications

(8 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Batini et al defined a DQ methodology [ 6 ] as “a set of guidelines and techniques that define a rational process for assessing and improving DQ, starting from describing the input information for a given application context, defines a rational process to assess and improve the quality of data”. A framework is considered as a theory-building and practice-oriented tool [ 71 ], providing a structure for using QA theory and methods [ 72 , 73 ]. The terms DQ methodology and DQ framework are often used interchangeably in related research.…”

Section: Data Quality Management Techniques Reviewmentioning

confidence: 99%

Data Quality Management in the Internet of Things

Zhang

Jeong

Lee

2021

Sensors

View full text Add to dashboard Cite

Nowadays, IoT is being used in more and more application areas and the importance of IoT data quality is widely recognized by practitioners and researchers. The requirements for data and its quality vary from application to application or organization in different contexts. Many methodologies and frameworks include techniques for defining, assessing, and improving data quality. However, due to the diversity of requirements, it can be a challenge to choose the appropriate technique for the IoT system. This paper surveys data quality frameworks and methodologies for IoT data, and related international standards, comparing them in terms of data types, data quality definitions, dimensions and metrics, and the choice of assessment dimensions. The survey is intended to help narrow down the possible choices of IoT data quality management technique.

show abstract

Section: Data Quality Management Techniques Reviewmentioning

confidence: 99%

Data Quality Management in the Internet of Things

Zhang

Jeong

Lee

2021

Sensors

View full text Add to dashboard Cite

show abstract

“…The proposed algorithms were applied on Chicago Crimes dataset [14], which is a real data with real time updates (weekly updates), that includes a large number of records (6,882,009 records) and attributes with heterogeneous data types. Before applying the proposed framework and algorithms, the heterogeneity degree of the dataset was measured by applying some data quality metrics which correspond to these degrees as [15] follows (Table 1): The total number of records after preprocessing is 6,179,427 records. According to the achieved results of clustering, the optimal number of clusters [16] is 2 clusters.…”

Section: Resultsmentioning

confidence: 99%

A Solution for Handling Big Data Heterogeneity Problem

El-Din

Senousy

2021

Digital Transformation Technology

View full text Add to dashboard Cite

In the recent years, Big Data became a prominent tendency in many different fields. It refers to any data that are intricate and gigantic in volume, with tremendous velocity and vast variety, which transcend the processing ability of conventional tools. Heterogeneity is one of the eminent problems facing Big Data. Most of the data that are gathered from diverse resources are semi-structured and unstructured, which makes information extraction a difficult task. In this paper, a framework and algorithms are proposed to provide a solution that can help in handling the heterogeneity problem of big data through increasing the homogeneity and decreasing the heterogeneity of data to a certain degree or level for text formats. The proposed solution is implemented using R language, R AnalyticFlow software environment and Apache Spark which is one of the big data processing tools.

show abstract

“…This review highlighted the conceptual frameworks and the criteria list approach of Data Quality (DQ) assessment. On the other hand, this article dealt with data quality in theory and did not explain how to use Data Ops technology to solve the problem of data quality (Micic et al, 2017).…”

Section: Paper Finding Discussionmentioning

confidence: 99%

“…For this reason, we found the chosen articles according to the topic discussed and we classified these research topics into three main areas including data quality impact, technical solution in the database area and technical solution in the computer science area. Most focus has been given to the technical solution in data goodness impact and technical solution in the database area (Zellal and Zaouia, 2016;Micic et al, 2017;Abdellaoui et al, 2016;Serra and Marotta, 2016;Izham Jaya, 2019). However, just five examination articles focus on the quality of data effect.…”

Section: Data Qualitymentioning

confidence: 99%

A Systematic Literature Review for Implementing Data Ops in the Data Warehouse Lifecycle during the ETL Phase

Bahaa¹,

Eldemerdash²,

Fahmy³

2021

Journal of Computer Science

View full text Add to dashboard Cite

Nowadays, no one can deny the importance of Data Ware House (DWH) in all organizations. The most important components in Data Ware House (DWH) are the Extraction, Transformation, Loading (ETL) phase. Data cleaning is a basic piece of the transformation stage in Data Warehousing. This may affect critical activities such as data collection and decision-making in various organizations Data Ops is an evaluation technique of Dev Ops in the data domain. This study conducts a Systematic Literature Review (SLR) to assess the previous studies of data warehouses related to Data Ops efforts. This study collects 55 primary studies related to the detection of Data Scrubbing, Data Consistency, Data warehouse, Dev Ops and Data Ops and we have conducted a Systematic Literature Review (SLR). Based on these findings, we discuss many concerns related to the study of current approaches in terms of abstraction level, metrics used, implementation and validation. That is why the analysis covers the published efforts between 2016 and 2021 since Data Ops is a significantly new technique. The survey should cover only research that took plan in recent years. The result of the study observed that 29% of the studies focused on solving the importance of data quality in the data warehouse, 62% of them focused on related Dev Ops, only 9% focused on Data Ops techniques and no 0% survey on enhancing ETL phase with Data Ops. This SLR brings to the attention of the research community several opportunities for using Data Ops in future research and the nearly proposed model DW Ops.

show abstract

Towards a Data Quality Framework for Heterogeneous Data

Cited by 19 publications

References 32 publications

Data Quality Management in the Internet of Things

Data Quality Management in the Internet of Things

A Solution for Handling Big Data Heterogeneity Problem

A Systematic Literature Review for Implementing Data Ops in the Data Warehouse Lifecycle during the ETL Phase

Contact Info

Product

Resources

About