A DaQL to Monitor Data Quality in Machine Learning Applications

Ehrlinger, Lisa; Haunschmid, Verena; Palazzini, Davide; Lettner, Christian

doi:10.1007/978-3-030-27615-7_17

Cited by 28 publications

(17 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As data model influences data warehouse quality, the study in [34] used statistical and machine learning methods to predict the effect of structural metrics on the efficiency and effectiveness of a conceptual model. Nonetheless, the study in [35] presented a tool for continuously monitoring the quality of data to increase the prediction accuracy of machine learning models.…”

Section: ) Existing Computing Methods For Scrutinizing Engaged Human Resourcesmentioning

confidence: 99%

Human-Survey Interaction (HSI): A Study on Integrity of Human Data Collectors in a Mass-Scale Hajj Pilgrimage Survey

et al. 2021

View full text Add to dashboard Cite

Mass gatherings (such as Hajj/Umrah), owing to their immensity, often present a variety of difficulties to the attendees. To have a comprehensive understanding of the difficulties as well as their potential remedies, surveying a good number of attendees is unavoidable and this can be facilitated through engaging data collectors. A crucial part here is identifying the integrity of the data collectors, which is yet to be explored in the literature to the best of our knowledge. To address this gap, in this study, we first perform a mass-scale data collection over Hajj/Umrah pilgrims through online (n=236) and in-person (n=752) surveys, where we cover a substantial part (n=712) through paid data collectors (n=53). We critically investigate data collection activities of the data collectors through focused group discussions involving expert reviewers. We explore two computing techniques to unveil the integrity of the data collectors from two different perspectives. Our study finds out influential (religious and socio-geographical) aspects that impact the data collection process. Besides, we find that the collaborative participation of expert reviewers is obligatory to scrutinize the data collectors' integrity. Additionally, our explored computing techniques, namely conflict analysis and learning-based analysis, can identify up to 74% and 99% of the unreliable data collectors respectively. We observe that, although these computing-based filtering can indicate the integrity up to a certain level, human-in-the-loop is unavoidable for concluding on the integrity. To the best of our knowledge, this is the first study of its kind (i.e., integrity analysis of the data collectors) in the literature.

show abstract

Section: ) Existing Computing Methods For Scrutinizing Engaged Human Resourcesmentioning

confidence: 99%

Human-Survey Interaction (HSI): A Study on Integrity of Human Data Collectors in a Mass-Scale Hajj Pilgrimage Survey

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The curation task can be implemented manually or automated. Data standardization, de-duplication, and matching are examples of automated tasks [19]. The focus of this research lies in automated data curation, within particular anomaly detection.…”

Section: E Data Curationmentioning

confidence: 99%

“…Models that are derived from it will perform poorly and generate unreliable conclusions. According to [19], the key to high-quality ML are the three principles of data quality: prevention, detection, and correction. Anomaly detection aims to detect abnormal patterns deviating from the rest of the data, called anomalies or outliers [20].…”

Section: E Data Curationmentioning

confidence: 99%

Unsupervised Anomaly Detection in Data Quality Control

Poon

Farshidi

et al. 2021

2021 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Data is one of the most valuable assets of an organization and has a tremendous impact on its long-term success and decision-making processes. Typically, organizational data error and outlier detection processes perform manually and reactively, making them time-consuming and prone to human errors. Additionally, rich data types, unlabeled data, and increased volume have made such data more complex. Accordingly, an automated anomaly detection approach is required to improve data management and quality control processes. This study introduces an unsupervised anomaly detection approach based on models comparison, consensus learning, and a combination of rules of thumb with iterative hyper-parameter tuning to increase data quality. Furthermore, a domain expert is considered a human in the loop to evaluate and check the data quality and to judge the output of the unsupervised model. An experiment has been conducted to assess the proposed approach in the context of a case study. The experiment results confirm that the proposed approach can improve the quality of organizational data and facilitate anomaly detection processes.

show abstract

“…A technique for improving accuracy in the absence of trustworthy sources is pointed out by [35], where many cheap labels from noisy labelers are combined. Various data validation [36], [37], [38], [39], [40] frameworks have been developed to address challenges relating to completeness, consistency, and timeliness: Deequ, DuckDQ, DaQL, and TFX enable practitioners to encode their expectations about how data may look like. Automated data cleaning methods based on anomaly or novelty detection offer the option to remove inconsistent data tuples even without any expert knowledge [41], [42], [43].…”

Section: Datasetsmentioning

confidence: 99%

Measuring Data Quality for Dataset Selection in Offline Reinforcement Learning

Swazinna¹,

Udluft²,

Runkler³

2021

Preprint

View full text Add to dashboard Cite

Recently developed offline reinforcement learning algorithms have made it possible to learn policies directly from pre-collected datasets, giving rise to a new dilemma for practitioners: Since the performance the algorithms are able to deliver depends greatly on the dataset that is presented to them, practitioners need to pick the right dataset among the available ones. This problem has so far not been discussed in the corresponding literature. We discuss ideas how to select promising datasets and propose three very simple indicators: Estimated relative return improvement (ERI) and estimated action stochasticity (EAS), as well as a combination of the two (COI), and empirically show that despite their simplicity they can be very effectively used for dataset selection.

show abstract

A DaQL to Monitor Data Quality in Machine Learning Applications

Cited by 28 publications

References 6 publications

Human-Survey Interaction (HSI): A Study on Integrity of Human Data Collectors in a Mass-Scale Hajj Pilgrimage Survey

Human-Survey Interaction (HSI): A Study on Integrity of Human Data Collectors in a Mass-Scale Hajj Pilgrimage Survey

Unsupervised Anomaly Detection in Data Quality Control

Measuring Data Quality for Dataset Selection in Offline Reinforcement Learning

Contact Info

Product

Resources

About