Towards a Data Collection Quality Model for Big Data Applications

Abdallah, Mohammad; Hammad, Alaa M.; Alzyadat, Wael

doi:10.1007/978-3-031-04216-4_11

Cited by 5 publications

(1 citation statement)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Evaluation strategy Domain expert knowledge Unique process General evaluation A product perspective on total data quality management [26] × × Aimq: a methodology for information quality assessment [32] × × Automated sensor verification using outlier detection in the Internet of thing [36] × × Data filtering system to avoid total data distortion in IoT networking [37] × × The daquincis architecture: a platform for exchanging and improving data quality in cooperative information systems [34] × [27] has been widely accepted within database management systems [28], [29], and within big data and IoT systems [30]- [32]. One of its core advantages is that it emphasizes an iterative approach to data quality management.…”

Section: Research Workmentioning

confidence: 99%

End-to-End Data Quality Assessment Using Trust for Data Shared IoT Deployments

2022

View full text Add to dashboard Cite

Continued development of communication technologies has led to the widespread Internet of Things (IoT) integration into various domains, including health, manufacturing, automotive and precision agriculture. This has further led to the increased sharing of data amongst such domains to abet innovation. Most of these IoT deployments, however, are based on heterogeneous, pervasive sensors, which can lead to quality issues in the recorded data. This can lead to sharing of inaccurate or inconsistent data. There is a significant need to assess the quality of the collected data, should it be shared with multiple application domains, as inconsistencies in the data could have financial or health ramifications. This paper builds on the recent research on trust metrics and presents a framework to integrate such metrics into the IoT data cycle for real-time data quality assessment. Critically, this paper adopts a mechanism to facilitate end user parameterisation of a trust metric tailoring it's use in the framework. Trust is a well-established metric that has been used to determine the validity of a piece or source of data in crowd-sourced or other unreliable data collection techniques such as that in IoT.The paper further discusses how the trust based framework eliminates the requirement for a gold standard and provides visibility into data quality assessment throughout the big data model.To qualify the use of trust as a measure of quality, an experiment is conducted using data collected from an IoT deployment of sensors to measure air quality in which low cost sensors were co-located with a gold standard reference sensor. The calculated trust metric is compared with two well understood metrics for data quality, Root Mean Squares Error (RMSE) and Mean Absolute Error (MAE). A strong correlation between the trust metric and the comparison metrics shows that trust may be used as an indicative quality metric for data quality. The metric incorporates the additional benefit of its ability for use in low context scenarios, as opposed to RMSE and MAE, which require a reference for comparison.

show abstract