Prediction of bile duct injury after transarterial chemoembolization for hepatocellular carcinoma: Model establishment and verification

Jian-xi, Guo; Zhang, Xueying; Kong, Jing

doi:10.3389/fonc.2022.973045

Cited by 2 publications

(1 citation statement)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bootstrap is commonly the method of choice to calculate the distribution of these model metrics 2,12‐15 . As a simulation‐based method, a reader cannot rerun the simulation the same way they can back‐calculate from a computed t test 16‐20 .…”

Section: Introductionmentioning

confidence: 99%

Computation of the distribution of model accuracy statistics in machine learning: Comparison between analytically derived distributions and simulation‐based methods

Huang

2023

Health Science Reports

View full text Add to dashboard Cite

Background and Aims: All fields have seen an increase in machine-learning techniques. To accurately evaluate the efficacy of novel modeling methods, it is necessary to conduct a critical evaluation of the utilized model metrics, such as sensitivity, specificity, and area under the receiver operator characteristic curve (AUROC). For commonly used model metrics, we proposed the use of analytically derived distributions (ADDs) and compared it with simulation-based approaches.Methods: A retrospective cohort study was conducted using the England National Health Services Heart Disease Prediction Cohort. Four machine learning models (XGBoost, Random Forest, Artificial Neural Network, and Adaptive Boost) were used. The distribution of the model metrics and covariate gain statistics were empirically derived using boot-strap simulation (N = 10,000). The ADDs were created from analytic formulas from the covariates to describe the distribution of the model metrics and compared with those of bootstrap simulation.Results: XGBoost had the most optimal model having the highest AUROC and the highest aggregate score considering six other model metrics. Based on the Anderson-Darling test, the distribution of the model metrics created from bootstrap did not significantly deviate from a normal distribution. The variance created from the ADD led to smaller SDs than those derived from bootstrap simulation, whereas the rest of the distribution remained not statistically significantly different.Conclusions: ADD allows for cross study comparison of model metrics, which is usually done with bootstrapping that rely on simulations, which cannot be replicated by the reader.

show abstract

Section: Introductionmentioning

confidence: 99%