Do different decision-analytic modeling approaches produce different results? A systematic review of cross-validation studies

Abstract

When choosing a modeling approach for health economic evaluation, certain criteria are often considered (e.g., population resolution, interactivity, time advancement mechanism, resource constraints). However, whether these criteria and their associated modeling approach impacts results remain poorly understood. A systematic review was conducted to identify cross-validation studies (i.e., modeling a problem using different approaches with the same body of evidence) to offer insight on this topic. With respect to population resolution, reviewed studies suggested that both aggregate- and individual-level models will generate comparable results, although a practical trade-off exists between validity and feasibility. In terms of interactivity, infectious-disease models consistently showed that, depending on the assumptions regarding probability of disease exposure, dynamic and static models may produce dissimilar results with opposing policy recommendations. Empirical evidence on the remaining criteria is limited. Greater discussion will therefore be necessary to promote a deeper understanding of the benefits and limits to each modeling approach.

Systematic narrative review of decision frameworks to select the appropriate modelling approaches for health economic evaluations

“…A recent systematic review was published focused on cross-validation work in health economic models, evaluating the impact of structural features on the choice of the modelling approach [20]. Population resolution was found to have minimal impact empirically as both aggregate- and individual-level models generated nearly identical results.…”

“…individual-level models required fewer simplifying assumptions, thus increasing its face validity but at the expense of being more time- and data-intensive; and vice versa). In terms of the criterion of interactivity, infectious-disease models have consistently showed that, depending on the assumptions regarding the probability of disease exposure, dynamic and static models will produce dissimilar results and lead to opposing policy recommendations [20]. Further research in this area is still required as it may provide the evidence that is necessary to better guide the development of evidence-based decision frameworks.…”

Modelling the impact of physical activity on public health: A review and critique

“…Beside the suitability of the modelling framework, the way these methods are implemented in practice (i.e. what structural assumptions are made) can affect the validity of costeffectiveness findings 15 . Comparably to how the National Institute of Health and Care Excellence (NICE)NICE evaluates models included as part of manufacturers submissions 33 , previews attempts to critique the plausibility of key structural assumptions of the models have been made in other public health evaluation settings 10,16 .…”