CONTEXT Assessment in the workplace is important, but many evaluations have shown that assessor agreement and discrimination are poor. Training discussions suggest that assessors find conventional scales invalid. We evaluate scales constructed to reflect developing clinical sophistication and independence in parallel with conventional scales.METHODS A valid scale should reduce assessor disagreement and increase assessor discrimination. We compare conventional and construct-aligned scales used in parallel to assess approximately 2000 medical trainees by each of three methods of workplace-based assessment (WBA): the mini-clinical evaluation exercise (mini-CEX); the acute care assessment tool (ACAT), and the case-based discussion (CBD). We evaluate how scores reflect assessor disagreement (V j and V j*p ) and assessor discrimination (V p ), and we model reliability using generalisability theory.RESULTS In all three cases the conventional scale gave a performance similar to that in previous evaluations, but the construct-aligned scales substantially reduced assessor disagreement and substantially increased assessor discrimination. Reliability modelling shows that, using the new scales, the number of assessors required to achieve a generalisability coefficient ‡ 0.70 fell from six to three for the mini-CEX, from eight to three for the CBD, from 10 to nine for 'on-take' ACAT, and from 30 to 12 for 'post-take' ACAT. CONCLUSIONSThe results indicate that construct-aligned scales have greater utility, both because they are more reliable and because that reliability provides evidence of greater validity. There is also a wider implication: the disappointing reliability of existing WBA methods may reflect not assessors' differing assessments of performance, but, rather, different interpretations of poorly aligned scales. Scales aligned to the expertise of clinician-assessors and the developing independence of trainees may improve confidence in WBA.assessment
-This paper outlines the development and evaluation of the utility of workplace-based assessments in higher medical training: case-based discussion (CbD); the acute care assessment tool (ACAT); audit assessment; teaching observation and patient survey (PS). The study population included trainees in higher medical training (ST3؉ ؉) from physician specialties in the UK. The pilot consisted of a prospective study of the use of the new assessments using local study coordinators (LSCs) and volunteer trainees. In total, 169 LSCs were recruited and 134 trainees returned at least one assessment. The endof-pilot questionnaire was returned by 44 assessors and 57 trainees. Questionnaire data and qualitative feedback were used to evaluate the validity, impact and feasibility of the new tools. For adequate reliability (co-efficient 0.7) a total of 12 CbDs; three ACATs and 16 PS raters are required. There was evidence for the validity and positive educational impact of all the tools. There were difficulties with the feasibility of the PS. KEY WORDS: acute care assessment tool, audit assessment,
Formative assessment is widely accepted in Education circles as being crucial to promoting student learning and, since 2010, the UK General Medical Council (2010) has mandated its use in workplace-based clinical training for all new doctors. As a result, the Royal College of Radiologists (RCR) instituted a range of formative workplace-based assessments including the Radiology Direct Observation of Procedural Skills (Rad-DOPS), in which supervisors appraise trainees' performance in carrying out clinical procedures. This paper reports on the quality of the written feedback in 2,500 Rad-DOPS online feedback forms in addressing the aims of the new assessment approach. Random samples of 500 were selected from the first three years of the new assessment implementation: 2010-13, and from 2016-17. Using an appropriate coding frame, the feedback was analysed across the samples against key trainee attributes including stage of training and level of adjudged competence. Criteria for identifying high quality feedback were derived from the literature and a simplified form of QualitativeComparative Analysis (QCA) was used to identify the conditions associated with high quality feedback. An average of 97% of the assessments contained written feedback but the number of instances of high quality feedback was found to be exceedingly small at around 5%. The paper offers suggestions for making the feedback process more purposeful in achieving the aims of formative assessment.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.