Evaluating the impact of prediction models: lessons learned, challenges, and recommendations

Kappen, Teus H.; Klei, Wilton A. van; Wolfswinkel, Leo van; Kalkman, Cor J.; Vergouwe, Yvonne; Moons, Karel G.M.

doi:10.1186/s41512-018-0033-6

Cited by 136 publications

(163 citation statements)

References 63 publications

(99 reference statements)

Supporting

Mentioning

149

Contrasting

Order By: Relevance

“…Ideally, a prediction model will also have an impact study to assess whether the model improves clinical decision-making and patient health outcomes. 24 Impact studies also help identify factors (ease of use, acceptability) that can affect implementation in routine care. 2 Impact studies investigate whether the model improves clinical decision-making and patient health outcomes.…”

Section: Validation Face Validity and Clinical Impact Studies Of Thementioning

confidence: 99%

“…2 Impact studies investigate whether the model improves clinical decision-making and patient health outcomes. 24 Conducting impact studies is expensive and requires significant effort, which is why only a few models have them. 24 Although not common, models with impact studies are assuredly effective in decision-making.…”

Section: Validation Face Validity and Clinical Impact Studies Of Thementioning

confidence: 99%

“…24 Conducting impact studies is expensive and requires significant effort, which is why only a few models have them. 24 Although not common, models with impact studies are assuredly effective in decision-making. Models with external validation and impact studies are recommended in clinical practice, as a model with accurate prediction is not beneficial if it is not generalisable or does not change behaviour.…”

Section: Validation Face Validity and Clinical Impact Studies Of Thementioning

confidence: 99%

“…Models providing an explicit recommendation (eg reoccurrence of an event, treatment options, lifestyle modification) showed greater impact on clinical practice than models without a recommendation (providing predicted probability only), despite models without a recommendation allowing more room for clinicians to apply their clinical judgement. 24…”

Section: Accessible Models With Decision Support For Successful Implementioning

confidence: 99%

See 3 more Smart Citations

Precision health through prediction modelling: factors to consider before implementing a prediction model in clinical practice

Chowdhury

Turin

2020

J Prim Health Care

View full text Add to dashboard Cite

INTRODUCTION: Precision medical practice emphasises early detection, improved surveillance and prevention through targeted intervention. Prediction models can help identify high-risk individuals to be targeted for healthy behavioural changes or medical treatment to prevent disease development and assist both health professionals and patients to make informed decisions. Concerns exist regarding the adequacy, accuracy, validity and reliability of prediction models. AIM:The purpose of this study is to introduce readers to the basic concept of prediction modelling in precision health and recommend factors to consider before implementing a prediction model in clinical practice. METHODS:Prediction models developed maintaining proper process and with quality prediction and validation can be used in clinical practice to improve patient care. RESULTS:Aspects of prediction models that should be considered before implementation include: appropriateness of the model for the intended purpose; adequacy of the model; validation, face validity and clinical impact studies of the model; a parsimonious model with data easily measured in clinical settings; and easily accessible models with decision support for successful implementation. DISCUSSION:Choosing clinical prediction models requires cautious consideration and several practical factors before implementing a model in clinical practice.

show abstract

Section: Validation Face Validity and Clinical Impact Studies Of Thementioning

confidence: 99%

Section: Validation Face Validity and Clinical Impact Studies Of Thementioning

confidence: 99%

Section: Validation Face Validity and Clinical Impact Studies Of Thementioning

confidence: 99%

Section: Accessible Models With Decision Support For Successful Implementioning

confidence: 99%

See 2 more Smart Citations

Precision health through prediction modelling: factors to consider before implementing a prediction model in clinical practice

Chowdhury

Turin

2020

J Prim Health Care

View full text Add to dashboard Cite

show abstract

“…Further research could also externally validate the infant models (the equations in Sections 3.7, 4.6 and 6.6), and studies assessing the clinical impact the models would also be valuable. Research into clinical impact is important to assess whether implementing a model improves clinical decision making and patient outcomes (Kappen et al, 2018).…”

Section: Strengths Limitations and Directions For Future Researchmentioning

confidence: 99%

Diagnosing conductive dysfunction in infants using wideband acoustic immittance: development and validation of predictive models

Myers¹

View full text Add to dashboard Cite

Background. Infants with early onset of otitis media have greater risk of recurrent and chronic infections that can affect language and development. Diagnostic tools able to quickly and accurately identify middle ear pathology early in infancy could help to facilitate timely intervention for these children.Wideband acoustic immittance (WAI) is an emerging technology for assessing middle ear function with significant advantages over established clinical tests such as tympanometry. WAI does not require pressurization of the ear canal, and is a high-resolution test, measuring middle ear function over a wider frequency range than is possible with tympanometry. Preliminary studies in infants have shown promising results, but further research assessing the diagnostic performance of WAI is needed. Also, the large amount of data generated by WAI can make results difficult to interpret. Research into suitable quantitative techniques to analyse results is still in its infancy. Prediction models are an attractive method for analysis of multivariate data as they can provide individualized probabilities that an infant has middle ear dysfunction. A clinically useful prediction model must be able to accurately discriminate between normal ears and those with middle ear dysfunction, and be well calibrated (i.e., give accurate predictions). However, the number of variables generated by WAI can cause issues with overfitting when developing multivariate models. An overfitted model will accurately describe the data it was developed on, but is likely to perform poorly when applied to new samples. Some form of data reduction or penalization is therefore necessary when modelling WAI. Another issue with developing WAI models is that there are substantial maturational effects on WAI through infancy that need to be controlled for. This can be achieved by either developing models for specific age groups (e.g., a model specifically for neonates), or by controlling for the effect of age by including interactions between age and WAI variables in a model, or by only using developmentally stable regions as predictor variables.Objective. The aim of this work was to investigate the diagnostic performance of WAI in infants by developing predictive models. Data reduction strategies such as selecting predictors based on prior research and principal component analysis were used to increase the likelihood that models would generalize to new samples. The effect of age was initially accounted for by developing age-specific models for neonates, 6-month infants, and 12-month infants (Chapters 2, 3 and 4, respectively). Longitudinal developmental effects on WAI through infancy were then investigated in Chapter 5, and this knowledge was used to develop a model controlling for the effect of age through infancy (6 to 18 months) (Chapter 6, Study 1). The neonate model was assessed for generalizability by applying the model to results from a new sample of infants (Chapter 6, Study 2)ii Methods. Tympanometry, distortion product otoacoustic emissions (DPOAEs) and ...

show abstract

Development and Validation of a Prediction Model for Future Estimated Glomerular Filtration Rate in People With Type 2 Diabetes and Chronic Kidney Disease

et al. 2023

View full text Add to dashboard Cite

ImportanceType 2 diabetes increases the risk of progressive diabetic kidney disease, but reliable prediction tools that can be used in clinical practice and aid in patients’ understanding of disease progression are currently lacking.ObjectiveTo develop and externally validate a model to predict future trajectories in estimated glomerular filtration rate (eGFR) in adults with type 2 diabetes and chronic kidney disease using data from 3 European multinational cohorts.Design, Setting, and ParticipantsThis prognostic study used baseline and follow-up information collected between February 2010 and December 2019 from 3 prospective multinational cohort studies: PROVALID (Prospective Cohort Study in Patients with Type 2 Diabetes Mellitus for Validation of Biomarkers), GCKD (German Chronic Kidney Disease), and DIACORE (Diabetes Cohorte). A total of 4637 adult participants (aged 18-75 years) with type 2 diabetes and mildly to moderately impaired kidney function (baseline eGFR of ≥30 mL/min/1.73 m2) were included. Data were analyzed between June 30, 2021, and January 31, 2023.Main Outcomes and MeasuresThirteen variables readily available from routine clinical care visits (age, sex, body mass index; smoking status; hemoglobin A1c [mmol/mol and percentage]; hemoglobin, and serum cholesterol levels; mean arterial pressure, urinary albumin-creatinine ratio, and intake of glucose-lowering, blood-pressure lowering, or lipid-lowering medication) were selected as predictors. Repeated eGFR measurements at baseline and follow-up visits were used as the outcome. A linear mixed-effects model for repeated eGFR measurements at study entry up to the last recorded follow-up visit (up to 5 years after baseline) was fit and externally validated.ResultsAmong 4637 adults with type 2 diabetes and chronic kidney disease (mean [SD] age at baseline, 63.5 [9.1] years; 2680 men [57.8%]; all of White race), 3323 participants from the PROVALID and GCKD studies (mean [SD] age at baseline, 63.2 [9.3] years; 1864 men [56.1%]) were included in the model development cohort, and 1314 participants from the DIACORE study (mean [SD] age at baseline, 64.5 [8.3] years; 816 men [62.1%]) were included in the external validation cohort, with a mean (SD) follow-up of 5.0 (0.6) years. Updating the random coefficient estimates with baseline eGFR values yielded improved predictive performance, which was particularly evident in the visual inspection of the calibration curve (calibration slope at 5 years: 1.09; 95% CI, 1.04-1.15). The prediction model had good discrimination in the validation cohort, with the lowest C statistic at 5 years after baseline (0.79; 95% CI, 0.77-0.80). The model also had predictive accuracy, with an R2 ranging from 0.70 (95% CI, 0.63-0.76) at year 1 to 0.58 (95% CI, 0.53-0.63) at year 5.Conclusions and RelevanceIn this prognostic study, a reliable prediction model was developed and externally validated; the robust model was well calibrated and capable of predicting kidney function decline up to 5 years after baseline. The results and prediction model are publicly available in an accompanying web-based application, which may open the way for improved prediction of individual eGFR trajectories and disease progression.

show abstract

Evaluating the impact of prediction models: lessons learned, challenges, and recommendations

Cited by 136 publications

References 63 publications

Precision health through prediction modelling: factors to consider before implementing a prediction model in clinical practice

Precision health through prediction modelling: factors to consider before implementing a prediction model in clinical practice

Diagnosing conductive dysfunction in infants using wideband acoustic immittance: development and validation of predictive models

Development and Validation of a Prediction Model for Future Estimated Glomerular Filtration Rate in People With Type 2 Diabetes and Chronic Kidney Disease

Contact Info

Product

Resources

About