Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist

Moons, Karel G.M.; Groot, Joris A H de; Bouwmeester, Walter; Vergouwe, Yvonne; Mallett, Susan; Altman, Douglas G.; Reitsma, Johannes B.; Collins, Gary S.

doi:10.1371/journal.pmed.1001744

Cited by 1,108 publications

(1,207 citation statements)

References 143 publications

Supporting

Mentioning

1,201

Contrasting

Unclassified

Order By: Relevance

“…A recently published external validation study did not validate the models with the best performance measures [47]. The lack of external validation of these results leads to limited generalizability, as development data often leads to inaccurate predictions when applied to other individuals than the individuals in the original study [16]. Therefore, we strongly advocate an external validation and head-to-head comparison of all models that were identified in this systematic review.…”

Section: Discussionmentioning

confidence: 93%

“…The specifics of our research question, which was framed according to the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) guidance [16], are shown in Table 1. The results have been reported conforming to the PRISMA statement [see Additional file 1].…”

Section: Methodsmentioning

confidence: 99%

“…For critical appraisal of the individual studies, we used the recently published CHARMS checklist [16]. In summary, the following items of the CHARMS checklist were handled: study characteristics and participants; outcome to be predicted; candidate predictors (for development studies only); sample size and handling of missing data; model development (for development studies only); model evaluation; and results and interpretation.…”

Section: Assessment Of Methodsological Qualitymentioning

confidence: 99%

See 2 more Smart Citations

Prediction models for the risk of gestational diabetes: a systematic review

et al. 2017

Self Cite

View full text Add to dashboard Cite

Background: Numerous prediction models for gestational diabetes mellitus (GDM) have been developed, but their methodological quality is unknown. The objective is to systematically review all studies describing first-trimester prediction models for GDM and to assess their methodological quality. Methods: MEDLINE and EMBASE were searched until December 2014. Key words for GDM, first trimester of pregnancy, and prediction modeling studies were combined. Prediction models for GDM performed up to 14 weeks of gestation that only include routinely measured predictors were eligible. Data was extracted by the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS). Data on risk predictors and performance measures were also extracted. Each study was scored for risk of bias. Results: Our search yielded 7761 articles, of which 17 were eligible for review (14 development studies and 3 external validation studies). The definition and prevalence of GDM varied widely across studies. Maternal age and body mass index were the most common predictors. Discrimination was acceptable for all studies. Calibration was reported for four studies. Risk of bias for participant selection, predictor assessment, and outcome assessment was low in general. Moderate to high risk of bias was seen for the number of events, attrition, and analysis. Conclusions: Most studies showed moderate to low methodological quality, and few prediction models for GDM have been externally validated. External validation is recommended to enhance generalizability and assess their true value in clinical practice.

show abstract

Section: Discussionmentioning

confidence: 93%

Section: Methodsmentioning

confidence: 99%

Section: Assessment Of Methodsological Qualitymentioning

confidence: 99%

See 1 more Smart Citation

Prediction models for the risk of gestational diabetes: a systematic review

et al. 2017

Self Cite

View full text Add to dashboard Cite

show abstract

“…Risk of bias assessment of the four included studies according to CHARMS checklist 19. [Color figure can be viewed at http://wileyonlinelibrary.com]…”

Section: Resultsmentioning

confidence: 99%

Prediction models for the risk of spontaneous preterm birth based on maternal characteristics: a systematic review and independent external validation

Meertens

Montfort

Scheepers

et al. 2018

Acta Obstet Gynecol Scand

View full text Add to dashboard Cite

IntroductionPrediction models may contribute to personalized risk‐based management of women at high risk of spontaneous preterm delivery. Although prediction models are published frequently, often with promising results, external validation generally is lacking. We performed a systematic review of prediction models for the risk of spontaneous preterm birth based on routine clinical parameters. Additionally, we externally validated and evaluated the clinical potential of the models.Material and methodsPrediction models based on routinely collected maternal parameters obtainable during first 16 weeks of gestation were eligible for selection. Risk of bias was assessed according to the CHARMS guidelines. We validated the selected models in a Dutch multicenter prospective cohort study comprising 2614 unselected pregnant women. Information on predictors was obtained by a web‐based questionnaire. Predictive performance of the models was quantified by the area under the receiver operating characteristic curve (AUC) and calibration plots for the outcomes spontaneous preterm birth <37 weeks and <34 weeks of gestation. Clinical value was evaluated by means of decision curve analysis and calculating classification accuracy for different risk thresholds.ResultsFour studies describing five prediction models fulfilled the eligibility criteria. Risk of bias assessment revealed a moderate to high risk of bias in three studies. The AUC of the models ranged from 0.54 to 0.67 and from 0.56 to 0.70 for the outcomes spontaneous preterm birth <37 weeks and <34 weeks of gestation, respectively. A subanalysis showed that the models discriminated poorly (AUC 0.51–0.56) for nulliparous women. Although we recalibrated the models, two models retained evidence of overfitting. The decision curve analysis showed low clinical benefit for the best performing models.ConclusionsThis review revealed several reporting and methodological shortcomings of published prediction models for spontaneous preterm birth. Our external validation study indicated that none of the models had the ability to predict spontaneous preterm birth adequately in our population. Further improvement of prediction models, using recent knowledge about both model development and potential risk factors, is necessary to provide an added value in personalized risk assessment of spontaneous preterm birth.

show abstract

“…Prognostic models are intended "to assist clinicians with their prediction of a patient's future outcome and to enhance informed decision making with the patient" [4]. Predictions from these models should have optimal performance at the time that they are practically implemented-the "intended moment of using the model" [5]. The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis) statement recommends to clearly define when the predictors used in the development of the model were measured [6] and states that "all predictors should be measured before or at the study time origin and known at the intended moment the model is intended to be used" [7].…”

Section: Introductionmentioning

confidence: 99%

Prognosis research ideally should measure time-varying predictors at their intended moment of use

et al. 2017

View full text Add to dashboard Cite

Background: Prognosis research studies (e.g. those deriving prognostic models or examining potential predictors of outcome) often collect information on time-varying predictors after their intended moment of use, sometimes using a measurement method different to that which would be used. We aimed to illustrate how estimates of predictor-outcome associations and prognostic model performance obtained from such studies may differ to those at the earlier, intended moment of use. Methods: We analysed data from two primary care cohorts of patients consulting for non-inflammatory musculoskeletal conditions: the Prognostic Research Study (PROG-RES: n = 296, aged >50 years) and the Primary care Osteoarthritis Screening Trial (POST: n = 756, >45 years). Both cohorts had collected comparable information on a potentially important time-varying predictor (current pain intensity: 0-10 numerical rating scale), other predictors (age, gender, practice) and outcome (patient-perceived non-recovery at 6 months). Using logistic regression models, we compared the direction and magnitude of predictor-outcome associations and model performance measures under two scenarios: (i) current pain intensity ascertained by the treating general practitioner in the consultation (the intended moment of use) and (ii) current pain intensity ascertained by a questionnaire mailed several days after the consultation. Results: In both cohorts, the predictor-outcome association was substantially weaker for pain measured at the consultation (OR (95% CI): PROG-RES 1.06 (0.95, 1.18); POST 1.04 (0.96, 1.12)) than for pain measured in the questionnaire (PROG-RES 1.34 (1.20, 1.48); POST 1.26 (1.18, 1.34)). The c-statistic of the multivariable model was lower when pain was measured at the consultation (c-statistic (95% CI): PROG-RES 0.57 (0.51, 0.64); POST 0.66 (0. 62, 0.70)) than when pain was measured in the questionnaire 0.75); POST 0.72 (0.68, 0.76)), reflecting the lower OR for pain at the consultation. Conclusions: Prognostic research studies ideally should measure time-varying predictors at their intended moment of use and using the intended measurement method. Otherwise, they may produce substantially different estimates of predictor-outcome associations and model performance. Researchers should report when, how and where predictors were measured and identify any significant departures from their intended use that may limit the applicability of findings in practice.

show abstract

Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist

Abstract: Carl Moons and colleagues provide a checklist and background explanation for critically appraising and extracting data from systematic reviews of prognostic and diagnostic prediction modelling studies. Please see later in the article for the Editors' Summary

Cited by 1,108 publications

References 143 publications

Prediction models for the risk of gestational diabetes: a systematic review

Prediction models for the risk of gestational diabetes: a systematic review

Prediction models for the risk of spontaneous preterm birth based on maternal characteristics: a systematic review and independent external validation

Prognosis research ideally should measure time-varying predictors at their intended moment of use

Contact Info

Product

Resources

About