Applying Kane’s validity framework to a simulation based assessment of clinical competence

Abstract

Assessment of clinical competence is complex and inference based. Trustworthy and defensible assessment processes must have favourable evidence of validity, particularly where decisions are considered high stakes. We aimed to organize, collect and interpret validity evidence for a high stakes simulation based assessment strategy for certifying paramedics, using Kane's validity framework, which some report as challenging to implement. We describe our experience using the framework, identifying challenges, decisions points, interpretations and lessons learned. We considered data related to four inferences (scoring, generalization, extrapolation, implications) occurring during assessment and treated validity as a series of assumptions we must evaluate, resulting in several hypotheses and proposed analyses. We then interpreted our findings across the four inferences, judging if the evidence supported or refuted our proposed uses of the assessment data. Data evaluating "Scoring" included: (a) desirable tool characteristics, with acceptable inter-item correlations (b) strong item-total correlations (c) low error variance for items and raters, and (d) strong inter-rater reliability. Data evaluating "Generalizability" included: (a) a robust sampling strategy capturing the majority of relevant medical directives, skills and national competencies, and good overall and inter-station reliability. Data evaluating "Extrapolation" included: low correlations between assessment scores by dimension and clinical errors in practice. Data evaluating "Implications" included low error rates in practice. Interpreting our findings according to Kane's framework, we suggest the evidence for scoring, generalization and implications supports use of our simulation-based paramedic assessment strategy as a certifying exam; however, the extrapolation evidence was weak, suggesting exam scores did not predict clinical error rates. Our analysis represents a worked example others can follow when using Kane's validity framework to evaluate, and iteratively develop and refine assessment strategies.

Current Trends and Opportunities for Competency Assessment in Pharmacy Education–A Literature Review

“…Although there are examples of validated tools in pharmacy education [47], many have been validated based on psychometric properties alone. There is increasing understanding that the isolated use of such approaches to the validation of current trends in complex assessment models may give an incomplete evaluation of the quality of the assessment overall [80,113,114]. More work is needed to ensure evidence-based validation approaches with multiple sources of evidence are used to support the proposed score interpretations, including the type of evidence collected, depending on the assessment instrument in question, and its intended application.…”

Validity evidence for programmatic assessment in competency-based education

“…However, a first limitation of this study is that we did not investigate Kane’s scoring, extrapolation, or implementation inferences. Future research should focus on providing evidence to examine the validity argument in support of (or against) these inferences illustrating how observations relate to scores (i. e., scoring inference), and how assessment scores relate to performance in the real world (i. e., extrapolation inference) and how the high-stakes performance decision impacts the individual, the curriculum, and society (i. e., implication inference) [12].…”

“…Kane’s validity framework offers a structure to gather information regarding the quality of assessment procedures by treating validity as involving a series of four inferences drawn from observed scores and considers whether those inferences are justified [10, 11]. These four inferences, for which sufficient validity arguments must be provided, include scoring, generalization, extrapolation, and implications [12]. Each validity argument is underpinned by a coherent analysis of evidence in support of (or against) the interpretations of test score meaning [11, 13].…”

Augmented reality in medical education: a systematic review

“…9,16 These criteria were initially adopted to evaluate testing instruments in surgical training 16 ; more recently, they have been used to validate surgical simulators and their readiness for implementation in surgical curricula. 9,18,19 While other validity frameworks have been developed in recent years, 20,21 none have been as widely used in evaluating simulation technology in medical education. 9,16 In order to validate ARAs at any of the five stages, studies were required to either conduct formal validity assessments or demonstrate outcomes that directly aligned with the validity requirements delineated in Figure 1.…”