Class imbalance should not throw you off balance: Choosing the right classifiers and performance metrics for brain decoding with imbalanced data

Thölke, Philipp; Mantilla-Ramos, Yorguin-Jose; Abdelhedi, Hamza; Maschke, Charlotte; Dehgan, Arthur; Harel, Yann; Kemtur, Anirudha; Berrada, Loubna Mekki; Sahraoui, Myriam; Young, Tammy; Pépin, Antoine Bellemare; Khantour, Clara El; Landry, Mathieu; Pascarella, Annalisa; Hadid, Vanessa; Combrisson, Etienne; O’Byrne, Jordan; Jerbi, Karim

doi:10.1101/2022.07.18.500262

Cited by 4 publications

(2 citation statements)

References 79 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Accuracy is well-accepted measurement for evaluating the performance of a classification problem. However, for an imbalanced dataset, the use of accuracy as an effective indicator has been questioned recently by various authors [49][50][51][52][53] . Therefore, alternative evaluation metrics for assessing the effectiveness of ML models for imbalanced dataset were explored, as accuracy alone is not trustworthy.…”

Section: Resultsmentioning

confidence: 99%

Phase prediction and experimental realisation of a new high entropy alloy using machine learning

Singh

Katiyar

Goel

et al. 2023

Sci Rep

View full text Add to dashboard Cite

Nearly ~ 108 types of High entropy alloys (HEAs) can be developed from about 64 elements in the periodic table. A major challenge for materials scientists and metallurgists at this stage is to predict their crystal structure and, therefore, their mechanical properties to reduce experimental efforts, which are energy and time intensive. Through this paper, we show that it is possible to use machine learning (ML) in this arena for phase prediction to develop novel HEAs. We tested five robust algorithms namely, K-nearest neighbours (KNN), support vector machine (SVM), decision tree classifier (DTC), random forest classifier (RFC) and XGBoost (XGB) in their vanilla form (base models) on a large dataset screened specifically from experimental data concerning HEA fabrication using melting and casting manufacturing methods. This was necessary to avoid the discrepancy inherent with comparing HEAs obtained from different synthesis routes as it causes spurious effects while treating an imbalanced data—an erroneous practice we observed in the reported literature. We found that (i) RFC model predictions were more reliable in contrast to other models and (ii) the synthetic data augmentation is not a neat practice in materials science specially to develop HEAs, where it cannot assure phase information reliably. To substantiate our claim, we compared the vanilla RFC (V-RFC) model for original data (1200 datasets) with SMOTE-Tomek links augmented RFC (ST-RFC) model for the new datasets (1200 original + 192 generated = 1392 datasets). We found that although the ST-RFC model showed a higher average test accuracy of 92%, no significant breakthroughs were observed, when testing the number of correct and incorrect predictions using confusion matrix and ROC-AUC scores for individual phases. Based on our RFC model, we report the development of a new HEA (Ni25Cu18.75Fe25Co25Al6.25) exhibiting an FCC phase proving the robustness of our predictions.

show abstract

Section: Resultsmentioning

confidence: 99%

Phase prediction and experimental realisation of a new high entropy alloy using machine learning

Singh

Katiyar

Goel

et al. 2023

Sci Rep

View full text Add to dashboard Cite

show abstract

“…To address the imbalance in the dataset and minimise error, we utilised an over-sampling approach to balance the sample (91). This may introduce noise into the synthetic sample in the dataset, resulting in some level of bias remaining in the models (92). In this study, we also used an integrative approach and included multiple features in a machine learning model with the 5-fold cross-validation techniques which enabled us to evaluate the performance of the model on multiple subsets of the training set.…”

Section: Discussionmentioning

confidence: 99%

Using machine learning to evaluate the value of genetic liabilities in classification of hypertension within the UK Biobank

MacCarthy,

Pazoki

2024

Preprint

View full text Add to dashboard Cite

Background and objective: Hypertension increases the risk of cardiovascular diseases (CVD) such as stroke, heart attack, heart failure, and kidney disease, contributing to global disease burden and premature mortality. Previous studies have utilized statistical and machine learning techniques to develop hypertension prediction models. Only a few have included genetic liabilities and evaluated their predictive values. This study aimed to develop an effective hypertension prediction model and investigate the potential influence of genetic liability for risk factors linked to CVD on hypertension risk using Random Forest (RF) and Neural Network (NN). Materials and methods: The study included 244,718 participants of European ancestry. Genetic liabilities were constructed using previously identified genetic variants associated with various cardiovascular risk factors through genome-wide association studies (GWAS). The sample was randomly split into training and testing sets at a 70:30 ratio. We used RF and NN techniques to develop prediction models in the training set with or without feature selection. We evaluated the models' discrimination performance using the area under the curve (AUC), calibration, and net reclassification improvement in the testing set. Results: The models without genetic liabilities achieved AUCs of 0.70 and 0.72 using RF and NN methods, respectively. Adding genetic liabilities resulted in a modest improvement in the AUC for RF but not for NN. The best prediction model was achieved using RF (AUC =0.71, Spiegelhalter z score= 0.10, P-value= 0.92, calibration slope=0.99) constructed in stage two. Conclusion: Incorporating genetic factors in the model may provide a modest incremental value for hypertension prediction beyond baseline characteristics. Our study highlighted the importance of genetic liabilities for both total cholesterol and LDL within the same prediction model adds value to the classification of hypertension.

show abstract

Differential Patterns of Associations within Audiovisual Integration Networks in Children with ADHD

Zamanzadeh,

Pourhedayat,

Bakouie

et al. 2023

Preprint

View full text Add to dashboard Cite

Attention deficit hyperactivity disorder (ADHD) is a neurodevelopmental condition characterized by symptoms of inattention and impulsivity and has been linked to disruptions in functional brain connectivity and structural alterations in large-scale brain networks. While anomalies in sensory pathways have also been implicated in the pathogenesis of ADHD, exploration of sensory integration regions remains limited. In this study, we adopted an exploratory approach to investigate the connectivity profile of auditory-visual integration networks (AVIN) in children with ADHD and neurotypical controls, utilizing the ADHD-200 rs-fMRI dataset. In addition to network-based statistics (NBS) analysis, we expanded our exploration by extracting a diverse range of graph theoretical features. These features served as the foundation for our application of machine learning (ML) techniques, aiming to discern distinguishing patterns between the control group and children with ADHD. Given the significant class imbalance in the dataset, ensemble learning models like balanced random forest (BRF), XGBoost, and EasyEnsemble classifier (EEC) were employed, designed to cope with unbalanced class observations. Our findings revealed significant AVIN differences between ADHD individuals and neurotypical controls, enabling automated diagnosis with moderate accuracy. Notably, the XGBoost model demonstrated balanced sensitivity and specificity metrics, critical for diagnostic applications, providing valuable insights for potential clinical use.These findings offer further insights into ADHD’s neural underpinnings and high-light the potential diagnostic utility of AVIN measures, but the exploratory nature of the study underscores the need for future research to confirm and refine these findings with specific hypotheses and rigorous statistical controls.

show abstract

Class imbalance should not throw you off balance: Choosing the right classifiers and performance metrics for brain decoding with imbalanced data

Cited by 4 publications

References 79 publications

Phase prediction and experimental realisation of a new high entropy alloy using machine learning

Phase prediction and experimental realisation of a new high entropy alloy using machine learning

Using machine learning to evaluate the value of genetic liabilities in classification of hypertension within the UK Biobank

Differential Patterns of Associations within Audiovisual Integration Networks in Children with ADHD

Contact Info

Product

Resources

About