Replicating Health Economic Models: Firm Foundations or a House of Cards?

Abstract

Health economic evaluation is a framework for the comparative analysis of the incremental health gains and costs associated with competing decision alternatives. The process of developing health economic models is usually complex, financially expensive and time consuming. For these reasons, model development is sometimes based on previous model-based analyses: this endeavour is usually referred to as model replication. Such model replication activity may involve the comprehensive reproduction of an existing model or "borrowing" all or part of a previously developed model structure. Generally speaking, the replication of an existing model may require substantially less effort than developing a new de novo model by bypassing, or undertaking in only a perfunctory manner, certain aspects of model development such as the development of a complete conceptual model and/or comprehensive literature searching for model parameters. A further motivation for model replication may be to draw on the credibility or prestige of previous analyses which have been published and/or used to inform decisionmaking. The acceptability and appropriateness of replicating models depends on the decision-making context: there exists a trade-off between the "savings" afforded by model replication and the potential "costs" associated with reduced model credibility due to the omission of certain stages of model development. This paper provides an overview of the different levels of, and motivations for, replicating health economic models, and discusses the advantages, disadvantages and caveats associated with this type of modelling activity. Irrespective of whether replicated models should be considered appropriate or not, complete replicability is generally accepted as a desirable property of health economic models, as reflected in critical appraisal checklists and good practice guidelines. To this end, the feasibility of comprehensive model replication is explored empirically across a small number of recent case studies.Recommendations are put forward for improving reporting standards to enhance comprehensive model replicability. Key points for decision makers Model replicability is generally perceived to be an indicator of the quality of published models. Model replication is associated with both advantages and disadvantages. Replication may be quicker and less expensive than developing a de novo model, however model authors should be aware that these "savings" may impact upon the credibility of the model. Our pilot study indicates that even amongst a very small sample of studies, the majority of the models considered could not be fully replicated.

Learnings for Health Economics from the Early Stages of the COVID-19 Pandemic

“…On the transparency front, seldom will you find an HE presentation, paper, or material that does not include a model diagram and a table of inputs. Although it may not always be possible to perfectly recreate models from reported data, we can get fairly close [2]. This has not been the case for many of the COVID-19 models released to date.…”

<p>Modelling the Cost-Effectiveness of Indacaterol/Glycopyrronium versus Salmeterol/Fluticasone Using a Novel Markov Exacerbation-Based Approach</p>

“…Most of the inputs for our model are based on the parameters used in the previous microsimulation model. 18 This brings advantages in terms of allowing assessment of the consistency of the Markov and microsimulation analyses, but also has the inherent limitations associated with model replication, 43 including the assumption that the prior model represents an adequate approach to evaluating cost-effectiveness. Unlike the previous microsimulation model, 18 our Markov model did not integrate pneumonia costs and therefore less completely reflects the potential benefits of IND/GLY.…”

Replication of Published Health Economic Obesity Models: Assessment of Facilitators, Hurdles and Reproduction Success

“…Method replicability and reproduction of results, which in other disciplines are common criteria of adequate research reporting to assure scientific rigor, are gaining importance in the field of health economic modeling, and have been the subject of recent studies [1,2]. In the field of health economic modeling, the topics of research reporting, model transparency, and model quality have been commonly discussed and investigated in great detail; this is reflected in the availability and application of multiple quality and reporting standards for health economic assessments [3][4][5].…”

“…According to previously published research in the field of model replication, comprehensive replicability is generally perceived to be desirable in health economic models [1], but additional work is needed to understand how to improve model transparency and in turn increase the chances of successful result reproduction [2]. These existing publications state that further work is needed to better understand Health Research Health Technology Assessment (HTA) program and is presented in a full-length HTA report, including an appendix with the health economic model, published in Health Technology Assessment [14].…”

“…Therefore the objectives of our research were (1) to replicate published health economic models in obesity, (2) to compare the reproduction results to the original results, (3) to determine facilitators, hurdles, and challenges of the replication process and to assess the reproduction success measured by different definitions suggested by McManus et al 2019 [6], and finally (4) to suggest model replication reporting standards to enhance model reproducibility.…”