Cost evaluations in health professions education: a systematic review of methods and reporting quality

Abstract

Context High‐quality research into education costs can inform better decision making. Improvements to cost research can be guided by information about the research questions, methods and reporting of studies evaluating costs in health professions education (HPE). Our objective was to appraise the overall state of the field and evaluate temporal trends in the methods and reporting quality of cost evaluations in HPE research. Methods We searched the MEDLINE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), EMBASE, Business Source Complete and ERIC (Education Resources Information Centre) databases on 31 July 2017. To evaluate trends over time, we sampled research reports at 5‐year intervals (2001, 2006, 2011 and 2016). All original research studies in HPE that reported a cost outcome were included. The Medical Education Research Study Quality Instrument (MERSQI) and the BMJ economic checklist were used to appraise methodological and reporting quality, respectively. Trends in quality over time were analysed. Results A total of 78 studies were included, of which 16 were published in 2001, 15 in 2006, 20 in 2011 and 27 in 2016. The region most commonly represented was the USA (n = 43). The profession most commonly referred to was that of the physician (n = 46). The mean ± standard deviation (SD) MERSQI score was 10.9 ± 2.6 out of 18, with no significant change over time (p = 0.55). The mean ± SD BMJ score was 13.5 ± 7.1 out of 35, with no significant change over time (p = 0.39). A total of 49 (63%) studies stated a cost‐related research question, 23 (29%) stated the type of cost evaluation used, and 31 (40%) described the method of estimating resource quantities and unit costs. A total of 16 studies compared two or more interventions and reported both cost and learning outcomes. Conclusions The absolute number of cost evaluations in HPE is increasing. However, there are shortcomings in the quality of methodology and reporting, and these are not improving over time.

Increasing value in research: cost evaluations in health professions education

“…Nevertheless, in line with Foo et al. 's conclusions, we experienced challenges such as using the right nomenclature, choosing the correct methodology and putting a price on the value‐side of selection. The recently published Association for Medical Education in Europe guide on how to read studies on educational costs and some exemplary studies (e.g.…”

“…However, this information is often not available: studies on costs are extremely scarce in HPE and studies including value are even more rare. Furthermore, the systematic review by Foo and colleagues in this issue of Medical Education concluded that the quality of most of the available research is substandard and has not increased since the beginning of this century. They found that shortcomings in the cost‐evaluation literature are mostly related to the methodology and reporting specific to these analyses.…”

“…This is, at least in part, caused by a lack of expertise in performing economic evaluations within the field of HPE. In addition, Foo and colleagues show that very few studies reporting cost data in HPE represent full economic evaluations of educational activities, in which costs and value of two or more approaches are evaluated and contrasted . These findings mirror the practice of higher education institutes: decisions on which educational activities to invest in are often not fuelled by full empirical economic evaluations.…”

“…In an attempt to get the wagons running smoothly, research in the field of selection for medical school should move towards a more general understanding of validity. Importantly, the ‘consequences’ pillar of validity should be taken into account explicitly, and must include costs as well as value related to selection tools and procedures, preferably also in comparison with other tools or procedures, as in cost–benefit evaluations …”

“…Moreover, Foo et al. 's suggestion to involve experts in health or educational economics at the earliest possible stage is essential to improve the quality of cost‐evaluation research …”