An evaluation of programmatic assessment across health professions education using contribution analysis

Abstract

Introduction: Programmatic assessment is gaining traction in health professions education. Despite this popularity, educators continue to grapple with complex contextual factors that impact implementation and outcome attainment. We used contribution analysis, a theory-informed evaluation method, to understand mechanism underpinning successful implementation. Method: Applying the six steps of contribution analysis, we developed a postulated theory of change (ToC) and then conducted a qualitative study with programmatic assessment stakeholders (graduates n = 15, supervisors n = 32, faculty n = 19) from four Australian dietetic programs. These data were analysed using the Framework Analysis method and integrated with data derived from a literature review across health disciplines, to assemble contribution claims and story, and verify the ToC. Results: Impact pathways for programmatic assessment from inception to implementation, and contribution to outcomes were articulated in the ToC. Leaders drove implementation using compromise and worked with a design team to apply the versatile principles. All people required training and purposefully designed tools were implemented within an ideological aligned system. Re-orientation of responsibilities situated learners as leaders, contributing to a psychologically safe environment which promoted a growth mindset. Credible high-stakes progression decisions were enabled, people experienced less stress, and derived gratification from assessment. External factors (institutional and accreditation requirements) and threats (resource mismatch, ideological misalignment, and capabilities of the people) were identified. Discussion: Contribution analysis revealed mechanism that educators can apply to implement a contextually responsive programmatic assessment across diverse settings.