A Cautionary Tale for Machine Learning Design: why we Still Need Human-Assisted Big Data Analysis

Roccetti, Marco; Delnevo, Giovanni; Casini, Luca; Salomoni, Paola

doi:10.1007/s11036-020-01530-6

Cited by 36 publications

(26 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…So, an additional human involvement is required to produce APIs to adjust incorrect information produced by ML. Roccetti et al [ 4 ] used human experts’ knowledge by defining some semantics to select high-quality instances from the data set, to train neural networks on fifteen million water meter readings with the goal of increasing the accuracy of the ML prediction.…”

Section: Resultsmentioning

confidence: 99%

“…The success of ML methods in real-world applications is dependent on not only the design of the ML procedures but also the quality and semantics of the data used for training ML methods [ 4 ]. Data quality has an important role in the success of ML approaches in prediction.…”

Section: Resultsmentioning

confidence: 99%

“…In many cases, a lack of sufficient high-quality data usually results in unsatisfactory outputs. For example, in Roccetti et al [ 4 ], neural networks were trained on a huge database of water consumption measurements and were unsuccessful to predict test data with sufficient precision. While the ML methods used could only approximate the relationship between model inputs and model outputs.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Review on Human–AI Interaction in Machine Learning and Insights for Medical Applications

Maadi

Khorshidi

Aickelin

2021

IJERPH

View full text Add to dashboard Cite

Objective: To provide a human–Artificial Intelligence (AI) interaction review for Machine Learning (ML) applications to inform how to best combine both human domain expertise and computational power of ML methods. The review focuses on the medical field, as the medical ML application literature highlights a special necessity of medical experts collaborating with ML approaches. Methods: A scoping literature review is performed on Scopus and Google Scholar using the terms “human in the loop”, “human in the loop machine learning”, and “interactive machine learning”. Peer-reviewed papers published from 2015 to 2020 are included in our review. Results: We design four questions to investigate and describe human–AI interaction in ML applications. These questions are “Why should humans be in the loop?”, “Where does human–AI interaction occur in the ML processes?”, “Who are the humans in the loop?”, and “How do humans interact with ML in Human-In-the-Loop ML (HILML)?”. To answer the first question, we describe three main reasons regarding the importance of human involvement in ML applications. To address the second question, human–AI interaction is investigated in three main algorithmic stages: 1. data producing and pre-processing; 2. ML modelling; and 3. ML evaluation and refinement. The importance of the expertise level of the humans in human–AI interaction is described to answer the third question. The number of human interactions in HILML is grouped into three categories to address the fourth question. We conclude the paper by offering a discussion on open opportunities for future research in HILML.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Review on Human–AI Interaction in Machine Learning and Insights for Medical Applications

Maadi

Khorshidi

Aickelin

2021

IJERPH

View full text Add to dashboard Cite

show abstract

“…This large dataset spanned a period in time, from the beginning of 2014 to the end of 2018. To train our deep learning model, at the end of a long validation process which is described at length in [ 8 , 9 ], we decided to use a smaller dataset, comprised of just those water meter devices, with at least three valid numerical readings. This dataset contained exactly 17,714 devices; where 15.652 were non-defective ones, and the remaining 2.062 were defective .…”

Section: Dataset Description: Type Of Variables Deep Learning and Prmentioning

confidence: 99%

“…Along this line, in this paper we describe a deep learning design experience, where we had initially a trouble on developing an appropriate deep learning model able to detect failures in mechanical water meter devices, because we tried to train that model by merging together the numerical information relative to water consumption with some device descriptors based on categorical information, thus resulting into an explosion in data dimensionality, that soon determined a deterioration of the prediction accuracy [ 8 , 9 ]. After several unsuccessful experiments conducted with alternative methodologies that either permitted to reduce the data space dimensionality or employed more traditional machine learning algorithms, we changed the training strategy.…”

Section: Introductionmentioning

confidence: 99%

An alternative approach to dimension reduction for pareto distributed data: a case study

et al. 2021

Self Cite

View full text Add to dashboard Cite

Deep learning models are tools for data analysis suitable for approximating (non-linear) relationships among variables for the best prediction of an outcome. While these models can be used to answer many important questions, their utility is still harshly criticized, being extremely challenging to identify which data descriptors are the most adequate to represent a given specific phenomenon of interest. With a recent experience in the development of a deep learning model designed to detect failures in mechanical water meter devices, we have learnt that a sensible deterioration of the prediction accuracy can occur if one tries to train a deep learning model by adding specific device descriptors, based on categorical data. This can happen because of an excessive increase in the dimensions of the data, with a correspondent loss of statistical significance. After several unsuccessful experiments conducted with alternative methodologies that either permit to reduce the data space dimensionality or employ more traditional machine learning algorithms, we changed the training strategy, reconsidering that categorical data, in the light of a Pareto analysis. In essence, we used those categorical descriptors, not as an input on which to train our deep learning model, but as a tool to give a new shape to the dataset, based on the Pareto rule. With this data adjustment, we trained a more performative deep learning model able to detect defective water meter devices with a prediction accuracy in the range 87–90%, even in the presence of categorical descriptors.

show abstract

Machine Learning–Based Data Analysis

Deepika

Kalaiselvi

2021

Machine Learning Approach for Cloud Data Analytics in IoT

View full text Add to dashboard Cite

A Cautionary Tale for Machine Learning Design: why we Still Need Human-Assisted Big Data Analysis

Cited by 36 publications

References 20 publications

A Review on Human–AI Interaction in Machine Learning and Insights for Medical Applications

A Review on Human–AI Interaction in Machine Learning and Insights for Medical Applications

An alternative approach to dimension reduction for pareto distributed data: a case study

Machine Learning–Based Data Analysis

Contact Info

Product

Resources

About