Biased accuracy in multisite machine-learning studies due to incomplete removal of the effects of the site

Solanes, Aleix; Palau, Pol; Fortea, Lydia; Salvador, Raymond; González-Navarro, Laura; Llach, Cristian-Daniel; Valentí, Marc; Vieta, Eduard; Raduà, Joaquim

doi:10.1016/j.pscychresns.2021.111313

Cited by 14 publications

(16 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…harvard.edu/fswiki/CorticalParcellation, www.frontiersin.org/articles/ 10.3389/fnins.2012.00171/full#h12 and https://freesurfer.net/fswiki/ SubcorticalSegmentation). For UKB, the MRI data were residualised with respect to scanning site (Alfaro-Almagro et al, 2021;Solanes et al, 2021) using linear models. To remove poor-quality data likely due to subject motion, UKB participants with Euler numbers (Rosen et al, 2018) of ≥3 SDs from the mean were identified and excluded (N = 778; de Lange, Barth, et al, 2020).…”

Section: Mri Data Acquisition and Processingmentioning

confidence: 99%

“…When adjusting for age-bias using fit coefficients derived from a training set to correct the predictions in independent test sets, the results were highly comparable (Table4). To check for potential scanning site effects(Alfaro- Almagro et al, 2021;Solanes et al, 2021), we plotted the UKB delta distributions and calculated the correlation between predicted and true age (r) for each site separately. As shown in FigureS2, the results were similar across the three sites.3.2 | Effects of age range and sample sizeThe age-bias corrected r and R 2 values are generally larger for all models, and the corrected values decrease with a narrower age range.In this scenario, the prediction variance is similar across test sets, which is a result of the training set being held constant.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Mind the gap: Performance metric evaluation in brain‐age prediction

Lange

Anatürk

Rokicki

et al. 2022

Human Brain Mapping

View full text Add to dashboard Cite

Estimating age based on neuroimaging-derived data has become a popular approach to developing markers for brain integrity and health. While a variety of machinelearning algorithms can provide accurate predictions of age based on brain characteristics, there is significant variation in model accuracy reported across studies. We predicted age in two population-based datasets, and assessed the effects of age range, sample size and age-bias correction on the model performance metrics Pearson's correlation coefficient (r), the coefficient of determination (R 2 ), Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE). The results showed that these metrics vary considerably depending on cohort age range; r and R 2 values are lower when measured in samples with a narrower age range. RMSE and MAE are also lower in samples with a narrower age range due to smaller errors/brain age delta values when predictions are closer to the mean age of the group. Across subsets with different age ranges, performance metrics improve with increasing sample size. Performance metrics further vary depending on prediction variance as well as mean age difference between training and test sets, and age-bias corrected metrics indicate

show abstract

Section: Mri Data Acquisition and Processingmentioning

confidence: 99%

mentioning

confidence: 99%

Mind the gap: Performance metric evaluation in brain‐age prediction

Lange

Anatürk

Rokicki

et al. 2022

Human Brain Mapping

View full text Add to dashboard Cite

show abstract

“…A major concern in neuroimaging research is the effect of site on the generalizability of ML models (Dockes et al, 2021; Solanes et al, 2021). Sites may differ in terms of scanner infrastructure, acquisition protocols and neuroimaging feature extraction pipelines as well as sample composition.…”

Section: Discussionmentioning

confidence: 99%

Systematic Evaluation of Machine Learning Algorithms for Neuroanatomically-Based Age Prediction in Youth

Modabbernia

Whalley

Glahn

et al. 2021

Preprint

View full text Add to dashboard Cite

Application of machine learning algorithms to structural magnetic resonance imaging (sMRI) data has yielded behaviorally meaningful estimates of the biological age of the brain (brainage). The choice of the machine learning approach in estimating brain-age in children and adolescents is important because age-related brain changes in these age-groups are dynamic. However, the comparative performance of the multiple machine learning algorithms available has not been systematically appraised. To address this gap, the present study evaluated the accuracy (Mean Absolute Error; MAE) and computational efficiency of 21 machine learning algorithms using sMRI data from 2,105 typically developing individuals aged 5 to 22 years from five cohorts. The trained models were then tested in an independent holdout datasets, comprising 4,078 pre-adolescents (aged 9-10 years). The algorithms encompassed parametric and nonparametric, Bayesian, linear and nonlinear, tree-based, and kernel-based models. Sensitivity analyses were performed for parcellation scheme, number of neuroimaging input features, number of cross-validation folds, and sample size. The best performing algorithms were Extreme Gradient Boosting (MAE of 1.25 years for females and 1.57 years for males), Random Forest Regression (MAE of 1.23 years for females and 1.65 years for males) and Support Vector Regression with Radial Basis Function Kernel (MAE of 1.47 years for females and 1.72 years for males) which had acceptable and comparable computational efficiency. Findings of the present study could be used as a guide for optimizing methodology when quantifying age-related changes during development.HighlightsEnsemble-based algorithms performed best in predicting brain age during developmentSupport vector regression offers optimal prediction accuracy and computational costsA 400-parcel resolution provided the best accuracy and computational efficiency

show abstract

“…Therefore, these potential effects of the site must be very carefully controlled. Ignoring them may yield an inflated accuracy, even when the models do not really predict ( 76 , 82 ).…”

Section: Common Pitfalls and Errorsmentioning

confidence: 99%

“…Here is where a technique called Transfer Learning appears. This approach can extract insights obtained in large general-purpose datasets and use that information to improve small dataset model creation ( 82 ). This technique has already been tested to improve Alzheimer’s disease classification ( 92 ).…”

Section: Challenges and Latest Advancementsmentioning

confidence: 99%