Development of an Objective Scoring System for Measurement of Resident Performance on the Human Patient Simulator

Scavone, Barbara M.; Sproviero, Michele T.; McCarthy, Robert J.; Wong, Cynthia A.; Sullivan, John T.; Siddall, Viva J.; Wade, Leonard D.

doi:10.1097/00000542-200608000-00008

Cited by 87 publications

(53 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The scenarios were originally developed by the Israeli Board of Anesthesiology Examination Committee. 2,3,12,13 Faculty members from the Department of Anesthesiology at the University of Florida, assisted by education and simulation experts, translated the scenarios and assessment tools, with maximal adherence to the original script, 2,3 scenario protocol, language, and assessment tools. No changes were made in scoring, assessment, pass/fail determinations, orientation of residents, or the examination process itself (see A P P E N D I X E S 1 AND 2 for development and validation).…”

Section: Methodsmentioning

confidence: 99%

Construct Validity and Generalizability of Simulation-Based Objective Structured Clinical Examination Scenarios

Sidi¹,

Gravenstein²,

Lampotang³

2014

Journal of Graduate Medical Education

View full text Add to dashboard Cite

Background It is not known if construct-related validity (progression of scores with different levels of training) and generalizability of Objective Structured Clinical Examination (OSCE) scenarios previously used with non-US graduating anesthesiology residents translate to a US training program. Objective We assessed for progression of scores with training for a validated high-stakes simulation-based anesthesiology examination. Methods Fifty US anesthesiology residents in postgraduate years (PGYs) 2 to 4 were evaluated in operating room, trauma, and resuscitation scenarios developed for and used in a high-stakes Israeli Anesthesiology Board examination, requiring a score of 70% on the checklist for passing (including all critical items). Results The OSCE error rate was lower for PGY-4 than PGY-2 residents in each field, and for most scenarios within each field. The critical item error rate was significantly lower for PGY-4 than PGY-3 residents in operating room scenarios, and for PGY-4 than PGY-2 residents in resuscitation scenarios. The final pass rate was significantly higher for PGY-3 and PGY-4 than PGY-2 residents in operating room scenarios, and also was significantly higher for PGY-4 than PGY-2 residents overall. PGY-4 residents had a better error rate, total scenarios score, general evaluation score, critical items error rate, and final pass rate than PGY-2 residents. Conclusions The comparable error rates, performance grades, and pass rates for US PGY-4 and non-US (Israeli) graduating (PGY-4 equivalent) residents, and the progression of scores among US residents with training level, demonstrate the construct-related validity and generalizability of these high-stakes OSCE scenarios.

show abstract

Section: Methodsmentioning

confidence: 99%

Construct Validity and Generalizability of Simulation-Based Objective Structured Clinical Examination Scenarios

Sidi¹,

Gravenstein²,

Lampotang³

2014

Journal of Graduate Medical Education

View full text Add to dashboard Cite

show abstract

“…Eighteen publications were finally selected. The retained articles were classified according to their main focus as NTSs, 8,14,[20][21][22][23][24] Medical Expert competency, 16,17,[25][26][27][28][29][30] or both. 13,15,31 The assessment tools were classified as checklists and global rating scales (GRS) (Fig.…”

Section: Development Of the Toolmentioning

confidence: 99%

“GIOSAT”: a tool to assess CanMEDS competencies during simulated crises

Neira

Bould

Nakajima

et al. 2013

Can J Anesth/J Can Anesth

View full text Add to dashboard Cite

“…Previous studies included between five and 28 experts in the Delphi process. 8,9,10 We achieved good internal consistency providing a high degree of confidence in our sample size of 10 experts.To address the issue of content validity we aimed to achieve consensus amongst a group of experts using a Delphi process. The concordance achieved was 0.668.…”

mentioning

confidence: 67%

The development of a scoring tool for the measurement of performance in managing hypotension and intra-operative cardiac arrest during spinal anaesthesia for caesarean section

Horsten

Wise

Ramroop

et al. 2015

Southern African Journal of Anaesthesia and Analgesia

View full text Add to dashboard Cite

Consortium, Cleveland Clinic, Cleveland, OH, USA *Corresponding author, email: garth.horsten@gmail.com Background: At level one hospitals in South Africa a high annual number of maternal deaths occur due to the unrecognised/ untreated complications of spinal anaesthesia. The authors developed a clinical scenario and scoring system to measure intern performance in managing hypotension and cardiac arrest during spinal anaesthesia for caesarean section on a human patient simulator. This system was then subjected to tests of validity and reliability. Methods: The simulator-based clinical scenario was developed by two specialist anaesthesiologists. A modified Delphi technique was used to achieve consensus among 10 anaesthetic specialists regarding a standardised scoring system. A total of 20 medical officers with a Diploma in Anaesthesiology and 20 interns completed the scenario and were scored by two senior anaesthesiologists. Results: Medical officers scored an average of 252 and 246 points, whereas interns scored an average of 216 and 215 points (p = 0.005 and p = 0.013, respectively). The scoring instrument demonstrated high inter-assessor reliability with an intra-class correlation coefficient of 0.983. Conclusions:The scoring tool was shown to be valid and reliable. It offers a standardised assessment process and may be used to refine institutional intern training programmes, with a view to improving anaesthesia skills in community service medical officers.Keywords: anaesthesia spinal, caesarean section, internship, residency, simulation, simulator BackgroundSouth Africa has a high annual number of anaesthetic maternal deaths. The majority of these occur at level one hospitals due to the unrecognised/untreated complications of spinal anaesthesia. 1These hospitals provide generalist services to inpatients and outpatients, who have ideally been referred from a community health centre or clinic.2 Research has shown a lack of training, experience and supervision in anaesthesia at these hospitals, 3 and that community service doctors (CSMOs), in their first year of practice after internship training, are required to provide obstetric anaesthesia without supervision. 4 Research has previously indicated that internship training (a two-year training period of doctors who have completed their undergraduate medical degree but do not yet have a licence to practise medicine unsupervised) inadequately prepares CSMOs to fulfil this role.5 Assessment of internship training programmes and intern performance at the end of their training is therefore critical.Simulation assessments offer a means of assessing performance without the risk of direct patient harm. 6,7 However, for any assessment tool to have wider applicability it needs to be validated and reproducible. Previous studies have developed valid and reliable simulator-based assessment tools for the measurement of trainee doctor performance in providing anaesthesia in simulated environments.8,9 Such tools have not been developed within South Africa. The objectives of...

show abstract

Development of an Objective Scoring System for Measurement of Resident Performance on the Human Patient Simulator

Cited by 87 publications

References 16 publications

Construct Validity and Generalizability of Simulation-Based Objective Structured Clinical Examination Scenarios

Construct Validity and Generalizability of Simulation-Based Objective Structured Clinical Examination Scenarios

“GIOSAT”: a tool to assess CanMEDS competencies during simulated crises

The development of a scoring tool for the measurement of performance in managing hypotension and intra-operative cardiac arrest during spinal anaesthesia for caesarean section

Contact Info

Product

Resources

About