Author contributions: M.S. conceived the study question. T.Y.L. and M.S. performed the literature review. M.S., T.Y.L., and J.P. developed the analytic plan. T.Y.L. and J.P. developed the statistical analysis framework. T.Y.L. conducted the analyses. M.S. and J.P. supervised the study progress and provided regular feedback. T.Y.L. wrote the first draft of the manuscript. All authors critically revised the manuscript and approved the final copy.
The unit normal loss integral (UNLI) has wide applicability in decision analysis and risk modeling, including as a solution for computation of various value-of-information (VoI) metrics. However, one limitation of the UNLI has been that its closed-form solution is available for only 1 dimension and thus can only be used for comparisons involving 2 strategies (where it is applied to the scalar incremental net benefit). We derived a closed-form solution for the 2-dimensional UNLI by the integration by parts technique. This enables the extension of the UNLI method to 3-comparison problems. We implemented this approach in R as part of the predtools package ( https://github.com/resplab/predtools/ ) and verified the accuracy of this implementation via Monte Carlo simulations. A case study based on a 3-arm clinical trial was used as an example for VoI analysis. Methods based on the closed-form solutions for the UNLI can now be extended to 3-decision comparisons, taking a fraction of a second to compute and not being subject to Monte Carlo error. Highlights The unit normal loss integral (UNLI) is widely used in decision analysis and risk modeling, including in the computation of various value-of-information metrics, but its closed-form solution is only applicable to comparisons of 2 strategies. We derive a closed-form solution for 2-dimensional UNLI, extending the applicability of the UNLI to 3-strategy comparisons. Such closed-form computation takes only a fraction of a second and is free from simulation errors that affect the hitherto available methods. In addition to the relevance in 3-strategy model-based and data-driven decision analyses, a particular application is in risk prediction modeling, where the net benefit of a classifier should always be compared with 2 default strategies of treating none and treating all.
IntroductionSevere asthma is associated with a disproportionally high disease burden, including the risk of severe exacerbations. Accurate prediction of the risk of severe exacerbations may enable clinicians to tailor treatment plans to an individual patient. This study aims to develop and validate a novel risk prediction model for severe exacerbations in patients with severe asthma, and to examine the potential clinical utility of this tool.Methods and analysisThe target population is patients aged 18 years or older with severe asthma. Based on the data from the International Severe Asthma Registry (n=8925), a prediction model will be developed using a penalised, zero-inflated count model that predicts the rate or risk of exacerbation in the next 12 months. The risk prediction tool will be externally validated among patients with physician-assessed severe asthma in an international observational cohort, the NOVEL observational longiTudinal studY (n=1652). Validation will include examining model calibration (ie, the agreement between observed and predicted rates), model discrimination (ie, the extent to which the model can distinguish between high-risk and low-risk individuals) and the clinical utility at a range of risk thresholds.Ethics and disseminationThis study has obtained ethics approval from the Institutional Review Board of National University of Singapore (NUS-IRB-2021-877), the Anonymised Data Ethics and Protocol Transparency Committee (ADEPT1924) and the University of British Columbia (H22-01737). Results will be published in an international peer-reviewed journal.Trial registration numberEuropean Union electronic Register of Post-Authorisation Studies, EU PAS Register (EUPAS46088).
Background A previously developed risk prediction model needs to be validated before being used in a new population. The finite size of the validation sample entails that there is uncertainty around model performance. We apply value-of-information (VoI) methodology to quantify the consequence of uncertainty in terms of net benefit (NB). Methods We define the expected value of perfect information (EVPI) for model validation as the expected loss in NB due to not confidently knowing which of the alternative decisions confers the highest NB. We propose bootstrap-based and asymptotic methods for EVPI computations and conduct simulation studies to compare their performance. In a case study, we use the non-US subsets of a clinical trial as the development sample for predicting mortality after myocardial infarction and calculate the validation EVPI for the US subsample. Results The computation methods generated similar EVPI values in simulation studies. EVPI generally declined with larger samples. In the case study, at the prespecified threshold of 0.02, the best decision with current information would be to use the model, with an incremental NB of 0.0020 over treating all. At this threshold, the EVPI was 0.0005 (relative EVPI = 25%). When scaled to the annual number of heart attacks in the US, the expected NB loss due to uncertainty was equal to 400 true positives or 19,600 false positives, indicating the value of further model validation. Conclusion VoI methods can be applied to the NB calculated during external validation of clinical prediction models. While uncertainty does not directly affect the clinical implications of NB findings, validation EVPI provides an objective perspective to the need for further validation and can be reported alongside NB in external validation studies. Highlights External validation is a critical step when transporting a risk prediction model to a new setting, but the finite size of the validation sample creates uncertainty about the performance of the model. In decision theory, such uncertainty is associated with loss of net benefit because it can prevent one from identifying whether the use of the model is beneficial over alternative strategies. We define the expected value of perfect information for external validation as the expected loss in net benefit by not confidently knowing if the use of the model is net beneficial. The adoption of a model for a new population should be based on its expected net benefit; independently, value-of-information methods can be used to decide whether further validation studies are warranted.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.