An alternative approach to dimension reduction for pareto distributed data: a case study

Abstract

Deep learning models are tools for data analysis suitable for approximating (non-linear) relationships among variables for the best prediction of an outcome. While these models can be used to answer many important questions, their utility is still harshly criticized, being extremely challenging to identify which data descriptors are the most adequate to represent a given specific phenomenon of interest. With a recent experience in the development of a deep learning model designed to detect failures in mechanical water meter devices, we have learnt that a sensible deterioration of the prediction accuracy can occur if one tries to train a deep learning model by adding specific device descriptors, based on categorical data. This can happen because of an excessive increase in the dimensions of the data, with a correspondent loss of statistical significance. After several unsuccessful experiments conducted with alternative methodologies that either permit to reduce the data space dimensionality or employ more traditional machine learning algorithms, we changed the training strategy, reconsidering that categorical data, in the light of a Pareto analysis. In essence, we used those categorical descriptors, not as an input on which to train our deep learning model, but as a tool to give a new shape to the dataset, based on the Pareto rule. With this data adjustment, we trained a more performative deep learning model able to detect defective water meter devices with a prediction accuracy in the range 87–90%, even in the presence of categorical descriptors.

AI-boosted CRISPR-Cas13a and total internal reflection fluorescence microscopy system for SARS-CoV-2 detection

“…Data quality affects algorithm performance. The imbalance and bias in the data distribution could weaken the capability of algorithms (Roccetti et al, 2019;Roccetti et al, 2021;Strickland 2022). Effective image preprocessing methods should be developed to improve image quality and ensure the reliability of the dataset.…”

An ensemble-based approach for automated medical diagnosis of malaria using EfficientNet

“…Currently, several machine learning approaches and problem solutions in numerous fields have been studied [ 23 , 35 , 42 , 43 , 50 ]. Several real-time applications have been proposed using transfer learning approaches such as Natural Language Processing (NLP), Automatic Speech Recognition (ASR), and to solve computer vision problems.…”

Doctor XAvIer: Explainable Diagnosis on Physician-Patient Dialogues and XAI Evaluation

“…The intuition behind FAD curves is inspired by counterfactual explanations -which describes how the prediction of a model changes when the input is perturbed (Wachter et al, 2018) -and the Pareto principle -which states that for many situations, approximately 80% of the outcome is due to 20% of causes (the "vital few") (Pareto, 1964;Roccetti et al, 2021). If a feature attribution method accurately ranks the most important features for a certain prediction and the Pareto principle holds true, then cumulatively dropping the most important features in descending order should yield a smaller and smaller decrease in model performance for that prediction.…”