Five ways to get a grip on the drawbacks of logic models in program evaluation

Abstract

Logic models are perhaps the most widely used tools in program evaluation work. They provide reasonably straightforward, visual illustrations of plausible links between program activities and outcomes. Consequently, they are employed frequently in stakeholder engagement, communication, and evaluation project planning. However, their relative simplicity comes with multiple drawbacks that can compromise the integrity of evaluation studies. In this Black Ice article, we outline key considerations and provide practical strategies that can help those engaged in evaluation work to identify and mitigate the limitations of logic models.

Conceptualizing Educational Comparability in Distributed Health Professions Education: A Scoping Review

“…Consistent with this view, it is noteworthy that program logic models have been criticized for their simplistic representation of causal relationships and lack of ability to account for dynamic connections in complex environments. 41,42 Such a model's capacity to represent the literature on comparability of educational experiences thereby highlights the lack of focus on the underlying interactions among various curricular elements, individual factors, and contextual components to enable the achievement of learning outcomes for students. For example, in a study that concluded that comparability of learning experiences was achieved through the observation that there was little difference in exam grades among sites, the authors cautioned that students' own self-study could have driven their similarity rather than having learning experiences that were comparable.…”

Education decision making at medical schools: there must be a better way

“…Five ways to get a grip on the shortcomings of logic models in program evaluation 17 by Betty Onyura and team outlined some limitations to logic models-such as neglecting to identify adverse outcomes caused by the interventions. They provided five strategies-including recommending efforts to revisit and revise logic models as contexts changes-to help educators mitigate the shortcomings to logic models.…”

To prove or improve? Examining how paradoxical tensions shape evaluation practices in accreditation contexts

“…However, they have not been matched by a significant parallel evolution in the methodological diversity of evaluation studies in the field. 13 Indeed, many contemporary evaluation approaches 14,15 have yet to be adopted or gain solid traction in the field. 13 This hesitant evolution suggests that not only is emphasis of methodological renewal as the solution to expressed evaluation challenges inadequate, but also that such emphasis may also contribute to the underexamination of entrenched barriers to transforming evaluation practices.…”

“…13 Indeed, many contemporary evaluation approaches 14,15 have yet to be adopted or gain solid traction in the field. 13 This hesitant evolution suggests that not only is emphasis of methodological renewal as the solution to expressed evaluation challenges inadequate, but also that such emphasis may also contribute to the underexamination of entrenched barriers to transforming evaluation practices. Specifically, situating limitations to the conduct of evaluation as purely technical or methodological constrains the field's examination of evaluation as a complex social practice that is subject to a plurality of socio-political interests; such interests 4,[16][17][18] ; must be subjected to empirical scrutiny in order to meaningfully understand the constraints or tensions shaping evaluation practice.…”

“…Indeed, such calls for methodological expansion in evaluation inquiry in HPE have persisted through the past decade. However, they have not been matched by a significant parallel evolution in the methodological diversity of evaluation studies in the field 13 . Indeed, many contemporary evaluation approaches 14,15 have yet to be adopted or gain solid traction in the field 13 .…”