Modelling to extract more information from clinical trials data: On some roles for the bootstrap

Sauerbrei, Willi; Royston, Patrick

doi:10.1002/sim.2954

Cited by 15 publications

(18 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Evaluation of model development methods in terms of both variable selection and coefficient estimates often reveals very different models can be selected based on bootstrap resampling of the patient dataset [55,56]. Where intermediate steps are used in model development, such as testing interaction terms or collapsing categories of variables, it might not be practicable to validate all model building steps fully [57].…”

Section: Resultsmentioning

confidence: 99%

Reporting performance of prognostic models in cancer: a review

Mallett

Royston

Waters

et al. 2010

BMC Med

Self Cite

167

126

View full text Add to dashboard Cite

BackgroundAppropriate choice and use of prognostic models in clinical practice require the use of good methods for both model development, and for developing prognostic indices and risk groups from the models. In order to assess reliability and generalizability for use, models need to have been validated and measures of model performance reported. We reviewed published articles to assess the methods and reporting used to develop and evaluate performance of prognostic indices and risk groups from prognostic models.MethodsWe developed a systematic search string and identified articles from PubMed. Forty-seven articles were included that satisfied the following inclusion criteria: published in 2005; aiming to predict patient outcome; presenting new prognostic models in cancer with outcome time to an event and including a combination of at least two separate variables; and analysing data using multivariable analysis suitable for time to event data.ResultsIn 47 studies, Cox models were used in 94% (44), but the coefficients or hazard ratios for the variables in the final model were reported in only 72% (34). The reproducibility of the derived model was assessed in only 11% (5) of the articles. A prognostic index was developed from the model in 81% (38) of the articles, but researchers derived the prognostic index from the final prognostic model in only 34% (13) of the studies; different coefficients or variables from those in the final model were used in 50% (19) of models and the methods used were unclear in 16% (6) of the articles. Methods used to derive prognostic groups were also poor, with researchers not reporting the methods used in 39% (14 of 36) of the studies and data derived methods likely to bias estimates of differences between risk groups being used in 28% (10) of the studies. Validation of their models was reported in only 34% (16) of the studies. In 15 studies validation used data from the same population and in five studies from a different population. Including reports of validation with external data from publications up to four years following model development, external validation was attempted for only 21% (10) of models. Insufficient information was provided on the performance of models in terms of discrimination and calibration.ConclusionsMany published prognostic models have been developed using poor methods and many with poor reporting, both of which compromise the reliability and clinical relevance of models, prognostic indices and risk groups derived from them.

show abstract

Section: Resultsmentioning

confidence: 99%

Reporting performance of prognostic models in cancer: a review

Mallett

Royston

Waters

et al. 2010

BMC Med

Self Cite

167

126

View full text Add to dashboard Cite

show abstract

“…For further uses to extract more information from multivariable data see Sauerbrei and Royston (2007). Obviously, issues of model complexity and model stability deserve much more attention in practice, in both low-and high-dimensional data.…”

Section: Discussionmentioning

confidence: 99%

Stability Investigations of Multivariable Regression Models Derived from Low- and High-Dimensional Data

Sauerbrei

Boulesteix

Binder

2011

Journal of Biopharmaceutical Statistics

Self Cite

112

View full text Add to dashboard Cite

Multivariable regression models can link a potentially large number of variables to various kinds of outcomes, such as continuous, binary, or time-to-event endpoints. Selection of important variables and selection of the functional form for continuous covariates are key parts of building such models but are notoriously difficult due to several reasons. Caused by multicollinearity between predictors and a limited amount of information in the data, (in)stability can be a serious issue of models selected. For applications with a moderate number of variables, resampling-based techniques have been developed for diagnosing and improving multivariable regression models. Deriving models for high-dimensional molecular data has led to the need for adapting these techniques to settings where the number of variables is much larger than the number of observations. Three studies with a time-to-event outcome, of which one has high-dimensional data, are used to illustrate several techniques. Investigations at the covariate level and at the predictor level are seen to provide considerable insight into model stability and performance. While some areas are indicated where resampling techniques for model building still need further refinement, our case studies illustrate that these techniques can already be recommended for wider use.

show abstract

“…Many of the recommendations highlighted across the PROGRESS series (see supplementary table on bmj.com) are relevant. For example, integrated standards of design, analysis, and reporting should be developed across the stages of discovery, replication, and evaluation of factors that potentially predict differential treatment response [38][39][40][41][42][43] (recommendation 10 in supplementary table). Here we highlight four key areas, with recommendations for improvement.…”

Section: Recommendations For Improving Prognosis Research For Stratifmentioning

confidence: 99%

Prognosis research strategy (PROGRESS) 4: Stratified medicine research

Hingorani

Windt

Riley

et al. 2013

BMJ

394

348

View full text Add to dashboard Cite

In patients with a particular disease or health condition, stratified medicine seeks to identify those who will have the most clinical benefit or least harm from a specific treatment. In this article, the fourth in the PROGRESS series, the authors discuss why prognosis research should form a cornerstone of stratified medicine, especially in regard to the identification of factors that predict individual treatment response

show abstract

Modelling to extract more information from clinical trials data: On some roles for the bootstrap

Cited by 15 publications

References 30 publications

Reporting performance of prognostic models in cancer: a review

Reporting performance of prognostic models in cancer: a review

Stability Investigations of Multivariable Regression Models Derived from Low- and High-Dimensional Data

Prognosis research strategy (PROGRESS) 4: Stratified medicine research

Contact Info

Product

Resources

About