Useful to whom? Evaluation utilisation theory and boundaries for programme evaluation scope

Abstract

Context: Programme evaluation is perpetually mandated in health professions education. Correspondingly, there has been an expansion in prescriptive methodological guides about 'how' to engage in various best practices in evaluation. However, what has gained less attention is an examination of the 'value' that different stakeholders seek to gain from programme evaluation. Evaluation utilisation theory and research can help us understand the diversity in both the driving forces for and the impact of programme evaluation. Awareness of the heterogeneity of evaluation utilisation priorities has implications for evaluation practices, including both methodological choices and understanding of the impact of programme evaluation in our field. Methods: In this article, I expound on the concept of evaluation utilisation by drawing on evaluation theory and research. 'Evaluation utilisation' refers to the application of programme evaluation processes and findings to influence thinking and action. Herein, different forms of 'evaluation utilisation' (including instrumental, conceptual, process and persuasive (symbolic and legitimative) utilisation) are discussed, as well as the related concept of 'evaluation misuse'. Furthermore, how the prioritisation of different forms of 'evaluation utilisation' can influence the scope and impact of evaluation scholarship are also discussed. Conclusions: Programme evaluation is a form of inquiry that requires more than the exercise of robust methodological techniques. Rather, it necessitates attention to the, sometimes divergent, priorities of different stakeholder groups. Although there is scant research on evaluation practices in health professions education, evaluation utilisation theory can inform critical examination of evaluation practices and impact in our field. Critically, understanding this body of work can help inform those engaged in evaluation about what they are (or should be) prioritising when they conduct programme evaluation, and better align evaluation methodologies with their scholarly, curricular and administrative intentions. Implications for future research and high-quality, transparent evaluation scholarship are presented. How to cite this article: Onyura B. Useful to whom? Evaluation utilisation theory and boundaries for programme evaluation scope.

A decade of faculty development for health professions educators: lessons learned from the Macy Faculty Scholars Program

“…Consequently, the authors conducted a robust program evaluation of the MFSP which is detailed in this paper. Subsequently, this paper can serve as an example of a academically rigorous program evaluation utilizing both quantitative and qualitative methodology and as additional data for other health profession educational leaders seeking to justify, build, or evaluate other FD programs to demonstrate the value of investment in health professions educators [16][17][18].…”

Towards Ecological Evaluation of Online Courses: Aiming for Thick Description

“…Like assessment of students (see Hounsell 2021, this book), the evaluation of teaching and courses can be formative (for learning and development) or summative (for accreditation, ranking, continuation) or a mix of both. Beyond those broad purposes, there are potentially many others (see, for example, the 'evaluation utilisation terms' in Onyura 2020). Moreover, different interests in the quality of online postgraduate courses and teaching might be categorised as pedagogical, aspirational or commercial.…”

“…And we particularly want to highlight how these examples show that evaluation can be useful in different ways, especially for development. For detailed analysis of usefulness, see Onyura (2020) on evaluation utilisation and Patton (2010) on developmental evaluation.…”

Two sides of the same coin: Quality improvement and program evaluation in health professions education

“…44,45 This leads to a situation where evaluations fail to be used beyond a summative judgement about program effectiveness. [46][47][48] There are newer evaluation approaches that emphasize the importance that program evaluation findings lead to improvements in praxis. For instance, Patton's (2010) Developmental Evaluation approach focuses on rapid and ongoing program development.…”

“…Therefore, program evaluators typically isolate metrics of program effectiveness from contextual variables whereas those who conduct quality improvement efforts consider contextual variables as intrinsic to the system, playing a role in the uptake and eventual impact of an intervention 44,45 . This leads to a situation where evaluations fail to be used beyond a summative judgement about program effectiveness 46‐48 …”