Internal and external validation of predictive models: A simulation study of bias and precision in small samples

Steyerberg, Ewout W.; Bleeker, Sacha E.; Moll, Henriëtte A.; Grobbee, Diederick E.; Moons, Karel G.M.

doi:10.1016/s0895-4356(03)00047-7

Cited by 480 publications

(356 citation statements)

References 22 publications

Supporting

Mentioning

335

Contrasting

Unclassified

Order By: Relevance

“…The 0:632þ formula as proposed in Efron and Tibshirani (1997) for the misclassification error has been extended for the Brier score and the AUC, e.g. in Steyerberg et al (2001Steyerberg et al ( , 2003; Wehberg and Schumacher (2004); Gerds and Schumacher (2007);Jiang and Simon (2007).…”

Section: Internal Validationmentioning

confidence: 99%

The Performance of Risk Prediction Models

Gerds¹,

Cai²,

2008

View full text Add to dashboard Cite

SummaryFor medical decision making and patient information, predictions of future status variables play an important role. Risk prediction models can be derived with many different statistical approaches. To compare them, measures of predictive performance are derived from ROC methodology and from probability forecasting theory. These tools can be applied to assess single markers, multivariable regression models and complex model selection algorithms. This article provides a systematic review of the modern way of assessing risk prediction models. Particular attention is put on proper benchmarks and resampling techniques that are important for the interpretation of measured performance. All methods are illustrated with data from a clinical study in head and neck cancer patients.

show abstract

Section: Internal Validationmentioning

confidence: 99%

The Performance of Risk Prediction Models

Gerds¹,

Cai²,

2008

View full text Add to dashboard Cite

show abstract

“…25 A common problem in using prognostic systems is that the systems are not generalizable. 40 The systematic reviews of prognostic models for TBIs that have been published in the past 20 years have generated criticism regarding their clinical or methodological validities; 15,33 only a few studies have been subjected to extensive internal and external validations. 2,10,16,36,37,39,42 Few studies have analyzed the characteristics of patients who succumbed to early death after a TBI.…”

mentioning

confidence: 99%

“…The external validation of models is essential for general applicability. 40 However, few studies perform this type of validation. 10,12,16,36,37,39,42 As in those publications, we found a worse discriminative ability in the validation set, although our model performed better than other recently published studies (0.88).…”

mentioning

confidence: 99%

Validation of a prognostic score for early mortality in severe head injury cases

Gómez¹,

De-la-Cruz

Lora

et al. 2014

JNS

View full text Add to dashboard Cite

Object Traumatic brain injury (TBI) represents a large health and economic burden. Because of the inability of previous randomized controlled trials (RCTs) on TBI to demonstrate the expected benefit of reducing unfavorable outcomes, the IMPACT (International Mission on Prognosis and Analysis of Clinical Trials in TBI) and CRASH (Corticosteroid Randomisation After Significant Head Injury) studies provided new methods for performing prognostic studies of TBI. This study aimed to develop and externally validate a prognostic model for early death (within 48 hours). The secondary aim was to identify patients who were more likely to succumb to an early death to limit their inclusion in RCTs and to improve the efficiency of RCTs. Methods The derivation cohort was recruited at 1 center, Hospital 12 de Octubre, Madrid (1990–2003, 925 patients). The validation cohort was recruited in 2004–2006 from 7 study centers (374 patients). The eligible patients had suffered closed severe TBIs. The study outcome was early death (within 48 hours post-TBI). The predictors were selected using logistic regression modeling with bootstrapping techniques, and a penalized reduction was used. A risk score was developed based on the regression coefficients of the variables included in the final model. Results In the validation set, the final model showed a predictive ability of 50% (Nagelkerke R2), with an area under the receiver operating characteristic curve of 89% and an acceptable calibration (goodness-of-fit test, p = 0.32). The final model included 7 variables, and it was used to develop a risk score with a range from 0 to 20 points. Age provided 0, 1, 2, or 3 points depending on the age group; motor score provided 0 points, 2 (untestable), or 3 (no response); pupillary reactivity, 0, 2 (1 pupil reacted), or 6 (no pupil reacted); shock, 0 (no) or 2 (yes); subarachnoid hemorrhage, 0 or 1 (severe deposit); cisternal status, 0 or 3 (compressed/absent); and epidural hematoma, 0 (yes) or 2 (no). Based on the risk of early death estimated with the model, 4 risk of early death groups were established: low risk, sum score 0–3 (< 1% predicted mortality); moderate risk, sum score 4–8 (predicted mortality between 1% and 10%); high risk, sum score 9–12 (probability of early death between 10% and 50%); and very high risk, sum score 13–20 (early mortality probability > 50%). This score could be used for selecting patients for clinical studies. For example, if patients with very high risk scores were excluded from our study sample, the patients included (eligibility score < 13) would represent 80% of the original sample and only 23% of the patients who died early. Conclusions The combination of Glasgow Coma Scale score, CT scanning results, and secondary insult data into a prognostic score improved the prediction of early death and the classification of TBI patients.

show abstract

“…These serum lipid levels, adjusted for age, sex and body mass index (BMI) by multiple linear regression models, were then compared in current and non-smokers. The bootstrap method with 1000 replications was used to derive a 95% confidence interval (CI) using the bias corrected and accelerated (BCa) method, which was shown to be appropriate for small sample sizes because of its high internal and external validity (16). The effect of interactions between a current smoking habit and the AA genotype on LDL-C and HDL-C levels, adjusted for age, sex and BMI, was calculated, and the genotype distribution of the PTPN11 polymorphism was tested for Hardy-Weinberg equilibrium.…”

Section: Methodsmentioning

confidence: 99%

Modification of the effect of smoking on cholesterol in Japanese carriers of a PTPN11 polymorphism

Okada

Suzuki²,

Nishio

et al. 2008

Mol Med Report

View full text Add to dashboard Cite

Abstract.A widely-expressed protein tyrosine phosphatase, SHP-2, regulates apoB secretion by insulin. We hypothesised that a variation in the SHP-2 gene, PTPN11, may interact with smoking to influence serum lipid concentrations. The study comprised 794 subjects (278 males and 516 females), aged 39-88 years, who attended a health examination in Hokkaido, Japan in 2003. Subjects were genotyped for a G/A PTPN11 polymorphism (rs2301756). The mean serum LDL cholesterol, HDL cholesterol and triglyceride levels, stratified by genotype, were compared between current and non-(never or ex-) smokers. Among the AA genotype carriers, LDL cholesterol levels were significantly decreased in current vs. nonsmokers [mean ± SD, 101±34 mg/dl vs. 146±40 mg/dl; b (adjusted mean difference) = -56.0; 95% CI -102.8 to -9.3; p=0.019 after adjustment for age, sex and body mass index]. In contrast, HDL cholesterol levels were increased in current vs. non-smokers (66±5 mg/dl vs. 58±12 mg/dl; b=14.9; 95% CI 3.1 to 26.8; p=0.014 after adjustment). The interaction between a current smoking habit and the AA genotype had a significant effect on LDL and HDL cholesterol (LDL-C, p=0.039; HDL-C, p=0.020). These data suggest that cigarette smoking might alter the metabolism of cholesterol in Japanese PTPN11 AA genotype carriers. IntroductionCigarette smoking and an increased level of LDL cholesterol (LDL-C) or decreased level of HDL cholesterol (HDL-C) are risk factors for cardiovascular disease. Increases in LDL-C levels and decreases in HDL-C levels have been observed in smokers (1), and a positive multiplicative interaction between smoking and LDL-C levels in coronary heart disease incidence has been observed (2). Cigarette smoking is an oxidative stress that causes genetic damage (3) and can either up-or downregulate gene expression levels (4). Studies have tried to clarify the effect of the gene-smoking interaction on lipid concentrations (5,6), and have revealed that both genetic variants and environmental factors, including a current smoking habit, influence serum cholesterol levels (7).Src homology 2 domain-containing protein tyrosine phosphatase-2 (SHP-2) is a widely-expressed protein tyrosine phosphatase (8) that regulates both acute and adaptive control of apoB secretion by insulin (9-11). The PTPN11 gene, which encodes SHP-2, is on chromosome 12 and contains 16 exons. Jamshidi et al selected three tagging single nucleotide polymorphisms (SNPs), rs11066322, rs11066320 and rs2301756, and demonstrated in a Caucasian female population that PTPN11 rs11066322 was associated with apoB levels and rs11066320 with LDL-C levels (12). However, the NCBI Variation Database shows that, in the Japanese population, SNP rs11066320 does not exist or is very rare, while rs11066322 and rs2301756 are in nearly complete linkage disequilibrium with each other. We therefore selected rs2301756, (IMS-JST057927 in the Japan Single Nucleotide Polymorphism database), located in intron 3, 223-bp upstream of exon 4, as a candidate SNP influencing LDL-C levels among the ...

show abstract

Internal and external validation of predictive models: A simulation study of bias and precision in small samples

Cited by 480 publications

References 22 publications

The Performance of Risk Prediction Models

The Performance of Risk Prediction Models

Validation of a prognostic score for early mortality in severe head injury cases

Modification of the effect of smoking on cholesterol in Japanese carriers of a PTPN11 polymorphism

Contact Info

Product

Resources

About