Development of workplace‐based assessments of non‐technical skills in anaesthesia*

Abstract

SummaryNon-technical skills are recognised as crucial to good anaesthetic practice. We designed and evaluated a specialty-specific tool to assess non-technical aspects of trainee performance in theatre, based on a system previously found reliable in a recruitment setting. We compared inter-rater agreement (multir-ater kappa) for live assessments in theatre with that in a selection centre and a video-based rater training exercise. Twenty-seven trainees participated in the first in-theatre assessment round and 40 in the second. Round-1 scores had poor inter-rater agreement (mean kappa = 0.20) and low reliability (generalisability coefficient G = 0.50). A subsequent assessor training exercise showed good inter-rater agreement, (mean kappa = 0.79) but did not improve performance of the assessment tool when used in round 2 (mean kappa = 0.14, G = 0.42). Inter-rater agreement in two selection centres (mean kappa = 0.61 and 0.69) exceeded that found in theatre. Assessment tools that perform reliably in controlled settings may not do so in the workplace. The Tooke report on Modernising Medical Careers [1] highlighted the need for specialty training to focus on the acquisition of excellence, rather than competence alone. Recent editorials have attempted to define how excellence in professionalism and other domains manifest in the workplace and highlight the importance of non-technical skills [2,3]. Workplace-based assessments are an invaluable tool for assessing professional practice in a comprehensive and valid way; however, only the mini Clinical Evaluation Exercise (mini-CEX) and multi-source feedback amongst the assessment tools currently used in the UK attempt to assess nontechnical skills. In addition, these tools focus on the achievement of basic clinical competence and employ methods with questionable accuracy, reliability and validity [4]. The mini-CEX has been shown to have wide inter-rater variability that results in poor discrimination between anaesthetic trainees [5], that is exacerbated by the lack of performance benchmarking and behavioural descriptors on the marking sheet. Variable scoring leniency and the face-toface nature of the assessment may also contribute to inaccurate scores [5,6]. Studies have established the value of multi-source feedback in certain settings [7], although concerns have been raised about victimisation by multisource feedback raters [8]. These current workplace-based assessment tools have been described as stressful, timeconsuming, artificial and difficult to organise [8], and rely on immediate access to an electronic portfolio in many specialties. Large numbers of assessors are required for each trainee to achieve a reliable assessment, suggesting that the feasibility of their use in high-stakes assessment is low [9]. In addition, students are able to select individual cases, case difficulty and specific assessors, despite evidence that the relationship between the observer and student may adversely influence the validity of the assessment [6,10,11]. None of the tools described a...

Are Examiners’ Judgments in OSCE-Style Assessments Influenced by Contrast Effects?

“…Unfortunately, judgment-based assessments are susceptible to a raft of psychometric weaknesses 1-3 that have not been satisfactorily resolved through either reformulation 4,5 or training. [6][7][8] In response, recent research into "assessor cognition" has sought to understand cognitive factors responsible for these limitations. This work has established that assessors appear to possess relatively unique, idiosyncratic performance theories (personal beliefs about what constitutes good performance) 9,10 and may use their own clinical abilities as their frame of reference (rather than recognized assessment standards) when judging the performance of trainees.…”

Assessing levels of support for residents following adverse outcomes: A national survey of anesthesia residency programs in the United States

“…There is significant evidence suggesting that human factors and non-technical skills training and evaluation can and should be incorporated into training programs. (Fletcher et al 2002;Fletcher 2003;Flin et al 2010;Crossingham et al 2012;Russ et al 2013). Unfortunately, less than half of physicians feel well trained in non-technical skills and managing conflict (Ramirez et al 1996;Rama-Maceiras 2012).…”

The relative reliability of actively participating and passively observing raters in a simulation‐based assessment for selection to specialty training in anaesthesia

“…Anaesthetists have recognised the value of simulation‐based education while acknowledging the need for further work on assessment tools . Two of the barriers to more widespread use of simulation in high‐stakes assessment are the costs of simulation and the difficulties of recruiting raters .…”