Development and Reporting of Prediction Models: Guidance for Authors From Editors of Respiratory, Sleep, and Critical Care Journals

Abstract

Prediction models aim to use available data to predict a health state or outcome that has not yet been observed. Prediction is primarily relevant to clinical practice, but is also used in research, and administration. While prediction modeling involves estimating the relationship between patient factors and outcomes, it is distinct from casual inference. Prediction modeling thus requires unique considerations for development, validation, and updating. This document represents an effort from editors at 31 respiratory, sleep, and critical care medicine journals to consolidate contemporary best practices and recommendations related to prediction study design, conduct, and reporting. Herein, we address issues commonly encountered in submissions to our various journals. Key topics include considerations for selecting predictor variables, operationalizing variables, dealing with missing data, the importance of appropriate validation, model performance measures and their interpretation, and good reporting practices. Supplemental discussion covers emerging topics such as model fairness, competing risks, pitfalls of "modifiable risk factors", measurement error, and risk for bias. This guidance is not meant to be overly prescriptive; we acknowledge that every study is different, and no set of rules will fit all cases. Additional best practices can be found in the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines, to which we refer readers for further details.

Generalizable deep temporal models for predicting episodes of sudden hypotension in critically ill patients: a personalized approach

“…A common drawback of deep learning methods can be their relative opacity. However, recent guidelines for the development of predictive models in a critical care setting emphasize the utility in making an accurate prediction so long as it is understood that the model makes no claim about causation 25 . Regardless, interpretability is an area of active interest in the research community and may be considered in future work.…”

Predicting an Unfavourable Course in Patients Hospitalised for COVID-19: The PREDICT-COVID Study

“…Internal validation is not always carried out or clearly presented, and without a careful internal validation an over tting of the model to the data is to be expected explaining that their reported performance is probably optimistic [10]. Contrary to previously published models, the methodology used to develop the prediction model proposed herein follows the published recommendations concerning information on source data, the presentation of inclusion and exclusion criteria for the prospectively included population, the explanation of the judgment criterion, the management of missing data, the explanation of the model used, the methodology used for internal validation, and the model performance measures and their interpretations [16].…”

“…Anonymised data for patients who were tested positive for Sars-Cov2 and hospitalised in one of the thirteen hospitals of the Hospices Civils de Lyon (Lyon, France) were extracted from the Noso-Cor database REF 16 on May 20, 2020. The data extracted concerned the following.…”

Prognostic models for predicting in-hospital paediatric mortality in resource-limited countries: a systematic review

“…For instance, George et al 20 despite using Cox regression to develop their model, did not report the verification of proportional hazard assumption nor explore the possibility of competing risks as recommended. 33 Other regression assumptions, for example, multicollinearity was equally not reported. However, since backward elimination method disregards redundant variables, we inferred the satisfaction of multicollinearity assumption if this method was applied.…”

“…75 76 Categorising continuous model predictors is a common practice by researchers, however, this practice discards a lot of information and its assumptions are rarely clinically plausible. 33 Finally, there is a risk of overfitting if the model includes more predictors than the dataset can support. The ratio of the events (deaths) to the number of independent candidate predictors have been discussed extensively in methodological papers elsewhere 77 78 and it has been recommended that ratio of the EPV should be at least 10.…”