Simulator assessment of innate endovascular aptitude versus empirically correct performance

Abstract

Although inherently subjective, the MRS yields reproducible scores that correlate with endovascular experience and formal training. Experts and novices with extensive video game experience achieve short completion times, whereas high MRS scores are achieved only by formally trained subjects. Innate endovascular aptitude and empirically correct performance may be two separate things, and aptitude may be acquirable through (or identified by) extensive nonmedical video game experience.

Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence

“…These studies have demonstrated that the global rating scales tend to be more sensitive to levels of expertise (Regehr et al 1998;Goff et al 2002Goff et al , 2005Broe et al 2006), which is consistent with evidence in the OSCE literature regarding checklists and global ratings (Hodges et al 1999;Hodges and McIlroy 2003). Based on this evidence (in addition to the higher inter-rater reliabilities and inter-station correlations for the global rating scales), some investigators have opted to rely solely on the global rating when implementing an OSATS examination (Bann et al 2003;Datta et al 2004;Hance et al 2005;Hislop et al 2006;Leong et al 2008;Kassab et al 2011). This approach seems reasonable based on the limited data that are available, but we also point out that the inter-rater reliabilities for the checklists are generally substantial (and higher than those described for OSCEs).…”

“…Of the included studies, an OSATS for obstetrics and gynecology training has shown increasing scores after additional laboratory skills training VanBlaricom et al 2005) and prior training on an endovascular simulator is associated with subsequent higher OSATS ratings (Hislop et al 2006). While these studies demonstrate training improves performance in the simulation setting, support of the extrapolation argument would be stronger if the link between simulation-based performance and realworld proficiency had also been established.…”

“…Since it's original description, numerous investigators have made minor adaptations to the global rating scale, generally related to specific aspects of the technical skill assessed or the conditions of observation. The modifications include: removing the items ''knowledge of instruments'' and ''use of assistants'' (Broe et al 2006) or replacing these terms with other task-relevant phrases (Friedlich et al 2001;Dath et al 2004;Hance et al 2005); adding an overall performance rating (Dath et al 2004;Datta et al 2006;Hance et al 2005;Hislop et al 2006;Leong et al 2008;Willems et al 2009); or adding an overall product quality rating (Hance et al 2005;Datta et al 2006;Hislop et al 2006;Leong et al 2008). Each of these modifications aligned with the original intent of the global rating scale, and in each case the instrument retained the majority of items from the original.…”

“…To summarize briefly, evidence generally supports the checklist and global rating scales, although checklists must be created for each new task or station and thus vary in both content and quality with each implementation. The pass/fail categorizations have generated decidedly mixed validity evidence Goff et al 2001Goff et al , 2002Hislop et al 2006;Goff et al 2007;VanHeest et al 2012) and were used in fewer than half the studies; for these reasons we have elected not to further consider this part of the OSATS in this review.…”

Evaluating training for a simulated team in complex whole procedure simulations in the endovascular suite

“…This paper provides guidance on training medical students and vascular trainees as members of a simulated interventionalist team. different scenarios including those beyond their level of experience, prior to interventions performed on real patients (Dayal et al 2004;Aggarwal et al 2006;Hsu et al 2004;Van Herzeele et al 2007;Reznick & MacRae 2006;Patel et al 2006;Hislop et al 2006;Neequaye et al 2007a). There are several reasons for working with simulated teams.…”