Perspectives on validation of clinical predictive algorithms

Hond, Anne de; Shah, Vaibhavi; Kant, Ilse; Calster, Ben Van; Steyerberg, Ewout W.; Hernandez‐Boussard, Tina

doi:10.1038/s41746-023-00832-9

Cited by 17 publications

(4 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These results suggest that our radiomics model has internal validity and temporal generalizability, but does not reach geographic or domain generalizability. 49 …”

Section: Discussionmentioning

confidence: 99%

Computed Tomography Radiomics for Preoperative Prediction of Spread Through Air Spaces in the Early Stage of Surgically Resected Lung Adenocarcinomas

Suh,

Han,

Kwon

et al. 2024

Yonsei Med J

View full text Add to dashboard Cite

Purpose To assess the added value of radiomics models from preoperative chest CT in predicting the presence of spread through air spaces (STAS) in the early stage of surgically resected lung adenocarcinomas using multiple validation datasets. Materials and Methods This retrospective study included 550 early-stage surgically resected lung adenocarcinomas in 521 patients, classified into training, test, internal validation, and temporal validation sets (n=211, 90, 91, and 158, respectively). Radiomics features were extracted from the segmented tumors on preoperative chest CT, and a radiomics score (Rad-score) was calculated to predict the presence of STAS. Diagnostic performance of the conventional model and the combined model, based on a combination of conventional and radiomics features, for the diagnosis of the presence of STAS were compared using the area under the curve (AUC) of the receiver operating characteristic curve. Results Rad-score was significantly higher in the STAS-positive group compared to the STAS-negative group in the training, test, internal, and temporal validation sets. The performance of the combined model was significantly higher than that of the conventional model in the training set {AUC: 0.784 [95% confidence interval (CI): 0.722–0.846] vs. AUC: 0.815 (95% CI: 0.759–0.872), p =0.042}. In the temporal validation set, the combined model showed a significantly higher AUC than that of the conventional model ( p =0.001). The combined model showed a higher AUC than the conventional model in the test and internal validation sets, albeit with no statistical significance. Conclusion A quantitative CT radiomics model can assist in the non-invasive prediction of the presence of STAS in the early stage of lung adenocarcinomas.

show abstract

“…These results suggest that our radiomics model has internal validity and temporal generalizability, but does not reach geographic or domain generalizability. 49 …”

Section: Discussionmentioning

confidence: 99%

Computed Tomography Radiomics for Preoperative Prediction of Spread Through Air Spaces in the Early Stage of Surgically Resected Lung Adenocarcinomas

Suh,

Han,

Kwon

et al. 2024

Yonsei Med J

View full text Add to dashboard Cite

show abstract

“…Only one review included external validations of previously developed models 52 , however limited details of model performance were presented. External validation is necessary to ensure a model is both reproducible and generalisable 85 86 , bringing the usefulness of the models included in these reviews into question. The PROGRESS framework suggests that multiple external validation studies should be conducted using independent datasets from different locations.…”

Section: Discussionmentioning

confidence: 99%

Development and validation of risk prediction tools for pressure injury occurrence: An umbrella review

Hillier,

Scandrett,

Coombe

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Background Pressure injuries (PIs) place a substantial burden on healthcare systems worldwide. Risk stratification of those who are at risk of developing PIs allows preventive interventions to be focused on patients who are at the highest risk. The considerable number of risk assessment scales and prediction models available underscore the need for a thorough evaluation of their development, validation and clinical utility. Our objectives were to identify and describe available risk prediction tools for PI occurrence, their content and development and validation methods used. Methods The umbrella review was conducted according to Cochrane guidance. MEDLINE, Embase, CINAHL, EPISTEMONIKOS, Google Scholar and reference lists were searched to identify relevant systematic reviews. Risk of bias was assessed using adapted AMSTAR-2 criteria. Results were described narratively. All included reviews contributed to build a comprehensive list of risk prediction tools. Results We identified five systematic reviews describing the development and validation of risk prediction tools for pressure injuries, 16 that assessed the prognostic accuracy of the tools and 10 that assessed the clinical effectiveness. Of the five reviews of model development and validation, four included only machine learning models. One review included detail about external validation, and this was the only review to include model performance metrics. Where quality assessment was completed (3 out of 5 reviews), most prediction tools were assessed by review authors as being at high risk of bias and no tools were assessed as being at low risk of bias. Conclusions Available tools do not meet current standards for the development or reporting of risk prediction models. The majority of tools have not been externally validated. Standardised and rigorous approaches to risk prediction model development and validation are needed. Registration The protocol was registered on the Open Science Framework (https://osf.io/tepyk).

show abstract

“…Then, closer than that, recognizing the former should be based on three dimensions: the temporal clinical practice following a continually updated clinical guideline [ 57 ], the spatial variation of the patient demographics [ 58 – 60 ], and the infinitely varied efficiencies of hospital operational systems [ 61 , 62 ]. The first raises a requirement on the prospective validation of prediction models and endows the prediction model with a remarkable valid period [ 57 , 63 ].…”

Section: Represent Accurately and Integrate Into Clinical Practice Se...mentioning

confidence: 99%

Clinical data mining: challenges, opportunities, and recommendations for translational applications

Qiao,

Chen,

Qian

et al. 2024

J Transl Med

View full text Add to dashboard Cite

Clinical data mining of predictive models offers significant advantages for re-evaluating and leveraging large amounts of complex clinical real-world data and experimental comparison data for tasks such as risk stratification, diagnosis, classification, and survival prediction. However, its translational application is still limited. One challenge is that the proposed clinical requirements and data mining are not synchronized. Additionally, the exotic predictions of data mining are difficult to apply directly in local medical institutions. Hence, it is necessary to incisively review the translational application of clinical data mining, providing an analytical workflow for developing and validating prediction models to ensure the scientific validity of analytic workflows in response to clinical questions. This review systematically revisits the purpose, process, and principles of clinical data mining and discusses the key causes contributing to the detachment from practice and the misuse of model verification in developing predictive models for research. Based on this, we propose a niche-targeting framework of four principles: Clinical Contextual, Subgroup-Oriented, Confounder- and False Positive-Controlled (CSCF), to provide guidance for clinical data mining prior to the model's development in clinical settings. Eventually, it is hoped that this review can help guide future research and develop personalized predictive models to achieve the goal of discovering subgroups with varied remedial benefits or risks and ensuring that precision medicine can deliver its full potential.

show abstract

Perspectives on validation of clinical predictive algorithms

Cited by 17 publications

References 23 publications

Computed Tomography Radiomics for Preoperative Prediction of Spread Through Air Spaces in the Early Stage of Surgically Resected Lung Adenocarcinomas

Computed Tomography Radiomics for Preoperative Prediction of Spread Through Air Spaces in the Early Stage of Surgically Resected Lung Adenocarcinomas

Development and validation of risk prediction tools for pressure injury occurrence: An umbrella review

Clinical data mining: challenges, opportunities, and recommendations for translational applications

Contact Info

Product

Resources

About