The relative reliability of actively participating and passively observing raters in a simulation‐based assessment for selection to specialty training in anaesthesia

Abstract

SummarySelection to specialty training is a high-stakes assessment demanding valuable consultant time. In one initial entry level and two higher level anaesthesia selection centres, we investigated the feasibility of using staff participating in simulation scenarios, rather than observing consultants, to rate candidate performance. We compared participant and observer scores using four different outcomes: inter-rater reliability; score distributions; correlation of candidate rankings; and percentage of candidates whose selection might be affected by substituting participants' for observers' ratings. Inter-rater reliability between observers was good (correlation coefficient 0.73-0.96) but lower between participants (correlation coefficient 0.39-0.92), particularly at higher level where participants also rated candidates more favourably than did observers. Station rank orderings were strongly correlated between the rater groups at entry level (rho 0.81, p < 0.001) but weaker at the two higher level centres (rho 0.52, p = 0.018; rho 0.58, p = 0.001). Substituting participants' for observers' ratings had less effect once scores were combined with those from other selection centre stations. Selection decisions for 0-20% of candidates could have changed, depending on the numbers of training posts available. We conclude that using participating raters is feasible at initial entry level only.

How effective are selection methods in medical education? A systematic review

“…Implementing an SC as part of a process for selecting medical students may be logistically complex. It requires the recruitment and training of faculty raters, and ongoing collaboration among academic and professional institutions and experts in different operational aspects of the process (including simulation, evaluation and measurement) . Moreover, as SCs are based on a multi‐trait, multi‐method design, they may comprise a large number of elements in different combinations and orders, meaning that the process by which an SC is designed and administered may influence the utility of the method.…”

“…have shown that the SC method can be expensive compared with other selection methods (approximately US$300 per candidate) and represents a logistically complex option, although on balance they still advocate SCs for use in medical school selection. Roberts et al . investigated the feasibility of having health care staff participate in simulated scenarios as raters in order to minimise the human resources required to implement an SC.…”

“…However, staff participant ratings were different from those of trained assessors and failed to achieve adequate levels of inter‐rater reliability. Nonetheless, Roberts et al . concluded that it may be viable to use other health care staff rather than trained assessors for some but not all stations.…”

Utility of selection methods for specialist medical training: A BEME (best evidence medical education) systematic review: BEME guide no. 45

“…One article (Patterson et al 2014) was a qualitative study exploring competency models to improve uniformity and calibration of the overall process. Two articles described the Australian GP selection center process, two quantitatively (Roberts et al 2014;Patterson, Rowett, et al 2016) and one qualitatively (Burgess et al 2014), two describing a selection center approach into anesthetics training (Gale et al 2010;Roberts et al 2013), three describing the UK GP selection center approach (Mitchison 2009;Patterson, Baron, et al 2009;Patterson, Lievens, et al 2013) and a systematic review (Patterson, Knight, et al 2016).…”

Adapting simulation responses from judgment-based to analytic-based scores: A process model, case study, and empirical evaluation of managers’ responses among a sample of managers.

“…Regardless of whether ratings are done virtually or in person, there are costs associated with the logistics and labor for each iteration. Further, rigorous training and frequent calibration is needed to obtain reliable and valid ratings, requiring additional time and resources (Cash, Hamre, Pianta, & Myers, 2012; Lievens, 2001; Roberts, Gale, Sice, & Anderson, 2013; Sugita, 2012). In addition to the labor costs associated with the ongoing use of human assessors, an automated scoring system may be expensive to initially develop (Bell et al, 2008), but once in place, it can be used an unlimited number of times with no increase in operating expenses (Luck et al, 2006).…”