Constructing a Validity Map for a Workplace-Based Assessment System: Cross-Walking Messick and Kane

Abstract

Problem Health professions education has shifted to a competency-based paradigm in which many programs rely heavily on workplace-based assessment (WBA) to produce data for summative decisions about learners. However, WBAs are complex and require validity evidence beyond psychometric analysis. Here, the authors describe their use of a rhetorical argumentation process to develop a map of validity evidence for summative decisions in an entrustment-based WBA system. Approach To organize evidence, the authors cross-walked 2 contemporary validity frameworks, one that emphasizes sources of evidence (Messick) and another that stresses inferences in an argument (Kane). They constructed a validity map using 4 steps: (1) Asking critical questions about the stated interpretation and use, (2) Seeking validity evidence as a response, (3) Categorizing evidence using both Messick’s and Kane’s frameworks, and (4) Building a visual representation of the collected and organized evidence. The authors used an iterative approach, adding new critical questions and evidence over time. Outcomes The first map draft produced 25 boxes of evidence that included all 5 sources of evidence detailed by Messick and spread across all 4 inferences described by Kane. The rhetorical question–response process allowed for structured critical appraisal of the WBA system, leading to the identification of evidentiary gaps. Next Steps Future map iterations will integrate evidence quality indicators and allow for deeper dives into the evidence. The authors intend to share their map with graduate medical education stakeholders (e.g., accreditors, institutional leaders, learners, patients) to understand if it adds value for evaluating their WBA programs’ validity arguments.

How argumentation theory can inform assessment validity: A critical review

“…HPE scholars have published robust validity evidence 13 , 38 , 39 , 40 , 106 for various assessment decisions organised within the commonly used frameworks of Messick 5 or Kane. 27 Although these provide excellent examples of how to organise validity evidence, they do not explicitly describe who should evaluate the evidence (audience), what structure the argument (not just the evidence) should take, and what criteria should be used for evaluation.…”

“…The concept of validity as argument has existed and evolved for decades. 34 , 35 , 36 Although many HPE scholars operate using Messick's or Kane's conceptualization of validity argumentation, 1 , 3 , 26 , 37 , 38 , 39 , 40 , 41 , 42 , 43 scholars outside HPE present other approaches. 44 Bachman advocates validation work should entail an assessment use argument , which is comprised of an assessment validity argument linking assessment performance to an interpretation and an assessment utilisation argument linking interpretations to a decision.…”

Reconstructing Neurath’s Ship: A Case Study in Reevaluating Equity in a Program of Assessment

“…Our program of assessment is centrally designed around the concept of having a defensible validity argument for decisions made using the data. 28 We have only begun investigating evidence of equity. 20 Are centrally designed and continuously monitored for evidence of equity, using contemporary theories of assessment and learning.…”

Advancing the assessment of clinical reasoning across the health professions: Definitional and methodologic recommendations

“…Where Messick's approach focuses on types of evidence, Kane focuses on the types of inference that are made moving from an assessment to judgment such as "competent." Thus, these two frameworks are complementary and have been combined in order to think comprehensively about the validity of assessment score interpretation [44].…”