Workplace-based assessments of junior doctors: do scores predict training difficulties?

Abstract

Medical Education 2011: 45: 1190–1198 Objectives Workplace‐based assessment (WPBA) is an increasingly important part of postgraduate medical training and its results may be used as evidence of professional competence. This study evaluates the ability of WPBA to distinguish UK Foundation Programme (FP) doctors with training difficulties and its effectiveness as a surrogate marker for deficiencies in professional competence. Methods We conducted a retrospective observational study using anonymised records for 1646 trainees in a single UK postgraduate deanery. Data for WPBAs conducted from August 2005 to April 2009 were extracted from the e‐portfolio database. These data included all scores submitted by trainees in FP years 1 and 2 on mini‐clinical evaluation exercise (mini‐CEX), case‐based discussion (CbD), direct observation of procedural skills (DOPS) and mini‐peer assessment tool (mini‐PAT) assessments. Records of trainees in difficulty, as identified by their educational supervisors, were tagged as index cases. Main outcome measures were odds ratios (ORs) for associations between mean WPBA scores and training difficulties. Further analyses by the reported aetiology of the training difficulty (health‐, conduct‐ or performance‐related) were performed. Results Of the 1646 trainees, 92 had been identified as being in difficulty. Mean CbD and mini‐CEX scores were lower for trainees in difficulty and an association was found between identified training difficulties and average scores on the mini‐CEX (OR = 0.54; p = 0.034) and CbD (OR = 0.39; p = 0.002). A receiver operator characteristic curve analysis of mean WPBA scores for diagnosing ‘in difficulty’ status yielded an area under the curve of 0.64, indicating weak predictive value. There was no statistical evidence that mean scores on DOPS and mini‐PAT assessments differed between the two groups. Conclusions Analysis of a large dataset of WPBA scores revealed significant associations between training difficulties and lower mean scores on both the mini‐CEX and CbD. Models show that using WPBA scores is, however, not a valid way of screening for trainees in difficulty. Workplace‐based assessments have value as formative assessments that prompt supervision, feedback and reflection. They should not be relied upon to certify competence and their use for such ends may reduce their effectiveness in training. Their results should be interpreted in the context of multiple other methods of assessment, with the aim of achieving a genuinely holistic and representative assessment of professional competence.

Using patients’ charts to assess medical trainees in the workplace: A systematic review

“…The relationship between the tool's content and the construct measured (content validity) was not evaluated in any of the studies. Three of the included articles examined the relationship of the assessment tool to other variables (Mitchell et al 2011;Brittlebank et al 2013;Eardley et al 2013). Mitchell et al (2011) concluded a significant association between the average score and training difficulties of all types (OR ¼ 3.9, 95% confidence interval [CI] ¼ 0.21-0.72).…”

“…Three of the included articles examined the relationship of the assessment tool to other variables (Mitchell et al 2011;Brittlebank et al 2013;Eardley et al 2013). Mitchell et al (2011) concluded a significant association between the average score and training difficulties of all types (OR ¼ 3.9, 95% confidence interval [CI] ¼ 0.21-0.72). Eardley et al (2013) found no evident relation between the stage of training and the summary rating given.…”

“…The remaining three articles were observational non-comparative studies (Mitchell et al 2011;Brittlebank et al 2013;Eardley et al 2013). Articles included studied postgraduate trainees who were enrolled in oncology (Jyothirmayi 2012), family medicine (Mitchell et al 2011), surgery (Eardley et al 2013), pediatric (Mehta et al 2013), psychiatry 61 arƟcles idenƟfied from electronic and hand searches 52 excluded: 32 unrelated 9 review arƟcles 6 evaluated pracƟcing physicians 5 evaluated allied health trainees 9 potenƟally relevant arƟcles for full text review 2 excluded: Addressed construct of the assessment tool 7 studies included in the review Figure 1. Summary of study selection process.…”

“…Only seven articles (Bodgener & Tavabie 2011;Johnson et al 2011;Mitchell et al 2011;Jyothirmayi 2012;Brittlebank et al 2013;Eardley et al 2013;Mehta et al 2013) were eligible for this review (Figure 1). …”

Validity and Feasibility of the Minicard Direct Observation Tool in 1 Training Program

“…1 Learner assessment in this approach generates rich narrative descriptions of the behaviors of a competent professional, which are used to determine progress and achievement toward these goals. The development of workplace-based assessments (WBAs) that are reliable, valid, feasible, educationally useful, and acceptable to faculty raters is critical to making these complex judgments.…”