Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales

Abstract

CONTEXT Assessment in the workplace is important, but many evaluations have shown that assessor agreement and discrimination are poor. Training discussions suggest that assessors find conventional scales invalid. We evaluate scales constructed to reflect developing clinical sophistication and independence in parallel with conventional scales.METHODS A valid scale should reduce assessor disagreement and increase assessor discrimination. We compare conventional and construct-aligned scales used in parallel to assess approximately 2000 medical trainees by each of three methods of workplace-based assessment (WBA): the mini-clinical evaluation exercise (mini-CEX); the acute care assessment tool (ACAT), and the case-based discussion (CBD). We evaluate how scores reflect assessor disagreement (V j and V j*p ) and assessor discrimination (V p ), and we model reliability using generalisability theory.RESULTS In all three cases the conventional scale gave a performance similar to that in previous evaluations, but the construct-aligned scales substantially reduced assessor disagreement and substantially increased assessor discrimination. Reliability modelling shows that, using the new scales, the number of assessors required to achieve a generalisability coefficient ‡ 0.70 fell from six to three for the mini-CEX, from eight to three for the CBD, from 10 to nine for 'on-take' ACAT, and from 30 to 12 for 'post-take' ACAT. CONCLUSIONSThe results indicate that construct-aligned scales have greater utility, both because they are more reliable and because that reliability provides evidence of greater validity. There is also a wider implication: the disappointing reliability of existing WBA methods may reflect not assessors' differing assessments of performance, but, rather, different interpretations of poorly aligned scales. Scales aligned to the expertise of clinician-assessors and the developing independence of trainees may improve confidence in WBA.assessment

Preliminary Validity Evidence for a Milestones-Based Rating Scale for Chart-Stimulated Recall

“…Our findings diverge from those of Crossley and colleagues, 7 who noted consistent and substantial variation between a ''construct-aligned'' assessment scale, analogous to our MBRS and a conventional scale. The way in which anchors in that study were developed was not described, and it is possible that their anchors were developed using a more detailed process.…”

“…A web-based platform was created for viewing and scoring. Formal rater training was purposefully not provided for 2 reasons: (1) lack of formal rater training simulates real-life settings where faculty development is not consistently provided, 7 and (2) MBRSs may improve scoring accuracy independent of rater training by offering a shared mental model of examinee performance. After being given a definition of CSR and guidance on navigating the online platform, all participants watched 3 videos in the same sequence during each session.…”

“…6 A large scale investigation in the United Kingdom found that a rating scale aligned to descriptions of behaviors improved reliability of ratings and reduced variability in scale interpretation. 7 Expert-derived frameworks of clinical competence, like the Accreditation Council for Graduate Medical Education (ACGME) Milestone Projects, may be helpful in defining level-specific performance for rating scales. 8,9 Although preliminary studies have shown advantages to using Milestones-Based Rating Scales (MBRSs) over SRSs in differentiating among levels of learners, 10,11 there is insufficient validity evidence to recommend conversion to MBRS for all assessments.…”

La utilización del Mini Clinical Evaluation Exercise (mini-CEX) en estudiantes de medicina

“…Norcini y Burch [13] sugerían que alrededor de cuatro encuentros eran sufi cientes en la mayoría de las situaciones clínicas, incluso aunque se requirieran 12-14 encuentros para alcanzar coefi cientes de fi abilidad de 0,8 [1,4,11]. Otro estudio, sin embargo, concluyó que una alineación de constructo de la escala del mini-CEX puede reducir signifi cativamente el número de evaluadores requerido para conseguir una buena fi abilidad [14]. Este enfoque puede reducir de seis a tres el número de evaluadores para alcanzar un coefi ciente de generalización igual o superior a 0,7.…”

Essential Facets of Competence That Enable Trust in Graduates: A Delphi Study Among Physician Educators in the Netherlands

“…Traditional checklists established for these purposes may not capture the essence of such entrustment decisions as they insufficiently align with this essential construct. 12 In clinical settings supervisors decide on a daily basis whether to trust a medical trainee with a specific task and to what extent supervision is needed. [13][14][15][16] Specific tasks pertain to a given procedure or skill that is to be carried out; general features pertain to task-independent characteristics.…”