Computation of the distribution of model accuracy statistics in machine learning: Comparison between analytically derived distributions and simulation‐based methods

Abstract

Background and Aims: All fields have seen an increase in machine-learning techniques. To accurately evaluate the efficacy of novel modeling methods, it is necessary to conduct a critical evaluation of the utilized model metrics, such as sensitivity, specificity, and area under the receiver operator characteristic curve (AUROC). For commonly used model metrics, we proposed the use of analytically derived distributions (ADDs) and compared it with simulation-based approaches.Methods: A retrospective cohort study was conducted using the England National Health Services Heart Disease Prediction Cohort. Four machine learning models (XGBoost, Random Forest, Artificial Neural Network, and Adaptive Boost) were used. The distribution of the model metrics and covariate gain statistics were empirically derived using boot-strap simulation (N = 10,000). The ADDs were created from analytic formulas from the covariates to describe the distribution of the model metrics and compared with those of bootstrap simulation.Results: XGBoost had the most optimal model having the highest AUROC and the highest aggregate score considering six other model metrics. Based on the Anderson-Darling test, the distribution of the model metrics created from bootstrap did not significantly deviate from a normal distribution. The variance created from the ADD led to smaller SDs than those derived from bootstrap simulation, whereas the rest of the distribution remained not statistically significantly different.Conclusions: ADD allows for cross study comparison of model metrics, which is usually done with bootstrapping that rely on simulations, which cannot be replicated by the reader.

Diabetes is associated with increased risk of death in COVID‐19 hospitalizations in Mexico 2020: A retrospective cohort study

“…In addition to overall survival, the restricted means survival time (RMST) was also calculated for diabetic and nondiabetic hospitalized COVID‐19 adult patients after propensity matching to control for confounders. This method fitted a parametric survival model to the data to estimate the mean survival time for the two groups 30 …”

Shapely additive values can effectively visualize pertinent covariates in machine learning when predicting hypertension

“…The model metrics were the Area under the Receiver Operator Characteristic Curve (AUROC), Sensitivity, Specificity, Positive Predictive Value, Negative Predictive Value, F1, Accuracy, and Balanced Accuracy. Additionally, the distribution of the Gain statistic, a measure of the percentage contribution of the variable to the model, for each covariate was assessed 34 . For model explanation, SHAP visualizations were performed for each independent covariate and visualized in figures.…”

Stochastic modeling of obesity status in United States adults using Markov Chains: A nationally representative analysis of population health data from 2017–2020

“…By incorporating uncertainty through bootstrap simulation, confidence intervals can be obtained, providing a measure of the reliability of the results. The combination of Markov chains and bootstrap simulation presents a robust methodology for modeling obesity, as it considers the probabilistic nature of weight transitions and accounts for data uncertainty 9 . This approach enhances the general understanding of the obesity epidemic and can inform evidence‐based interventions and policies aimed at addressing this significant public health issue.…”