Effect of rater training on the reliability of technical skill assessments: a randomized controlled trial

Robertson, Reagan L.; Vergis, Ashley; Gillman, Lawrence M.; Park, Jason

doi:10.1503/cjs.015917

Cited by 13 publications

(10 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous studies have demonstrated the importance of rater training and experience with assessment tools in improving interrater reliability. 25 , 26 , 27 Because future iterations of the Bootcamp employ the median sternotomy training model and procedure checklist, the interrater reliability can be improved with training of raters and improved iterations of the checklist. Additionally, analysis of checklist items with recurrent poor interrater reliability and discussion with raters to ascertain sources of discrepancy may help elucidate whether rater training is sufficient, or whether modifications to the checklist are needed.…”

Section: Discussionmentioning

confidence: 99%

Development of a median sternotomy simulation model for cardiac surgery training

Juanda

Ngu

et al. 2020

JTCVS Techniques

View full text Add to dashboard Cite

Objective: We sought to develop a simulation model to train resident physicians in the performance of a median sternotomy.Methods: A modified Delphi consensus process was used with cardiac surgery staff to develop a 20-point checklist for the safe performance of a median sternotomy. Thirteen junior cardiac surgery trainees from across Canada participated in this study to assess the simulation model. Trainees performed the sternotomy before and after reviewing an instructional video. Two senior cardiac surgery resident physicians assessed the participants with the checklist during each session. An entry and exit questionnaire was given to the participants to evaluate the simulation model. Results: Participants scored higher after the training (14.3 AE 2.0) compared with before training (8.0 AE 3.1) (P < .001). The mean duration of time for participants to complete the sternotomy was shorter before training (188 AE 52 seconds vs 228 AE 58 seconds; P ¼ .003). The checklist interrater reliability was k ¼ 0.47 (moderate) for before training and k ¼ 0.37 (fair) for after training. All study participants rated the simulation sessions as very useful or extremely useful. Conclusions: Using the simulation model, training video, and checklist, trainees were able to improve their skill in performing a median sternotomy. This improvement was associated with longer times to complete all procedure steps. Rater training may further improve interrater reliability. Our median sternotomy checklist and simulation model can be adopted for the technical skills training of future cardiac surgery trainees. (JTCVS Techniques 2020;2:109-16

show abstract

Section: Discussionmentioning

confidence: 99%

Development of a median sternotomy simulation model for cardiac surgery training

Juanda

Ngu

et al. 2020

JTCVS Techniques

View full text Add to dashboard Cite

show abstract

“…The VAS score is typically used to assess pain or anxiety among patients, but it can be also used for other purposes, e.g., among residents for assessing their own management in a special kind of anesthesia [21], and in assessing the overall quality of patient sign-out from the emergency department [22]. For trainees' surgical skills, the VAS score has been used to evaluate suturing and knot tying skills [23], showing the VAS score and the OSATS for global rating skills 'good' for educational purposes with interrater reliability (IRR) 0.71 in a group where assessors were trained to the use of scales. The IRR was slightly lower, though VAS scores correlated well with the combined OSATS score, and the scale was easy to use.…”

Section: Discussionmentioning

confidence: 99%

Simulator training and residents’ first laparoscopic hysterectomy: a randomized controlled trial

2019

View full text Add to dashboard Cite

Background Hysterectomy rates are decreasing in many countries, and virtual reality simulators bring new opportunities into residents' surgical education. The objective of this study was to evaluate the effect of training in laparoscopic hysterectomy module with virtual reality simulator on surgical outcomes among residents performing their first laparoscopic hysterectomy. Methods This randomized study was carried out at the Department of Obstetrics and Gynecology in Helsinki University Hospital and Hyvinkää Hospital. We recruited twenty residents and randomly signed half of them to train ten times with the laparoscopic hysterectomy module on a virtual reality simulator, while the rest represented the control group. Their first laparoscopic hysterectomy was video recorded and assessed later by using the Objective Structured Assessment of Technical Skills (OSATS) forms and Visual Analog Scale (VAS). The scores and surgical outcomes were compared between the groups. Results The mean OSATS score for the Global Rating Scale (GRS) was 17.0 (SD 3.1) in the intervention group and 11.2 (SD 2.4) in the control group (p = 0.002). The mean procedure-specific OSATS score was 20.0 (SD 3.3) and 16.0 (SD 2.8) (p = 0.012), and the mean VAS score was 55.0 (SD 14.8) and 29.9 (SD 14.9) (p = 0.001). Operative time was 144 min in the intervention group and 165 min in the control group, but the difference did not reach statistical significance (p = 0.205). There were no differences between the groups in blood loss or direct complications. Conclusion Residents training with a virtual reality simulator prior to the first laparoscopic hysterectomy seem to perform better in the actual live operation. Thus, a virtual reality simulator hysterectomy module could be considered as a part of laparoscopic training curriculum. Keywords Surgical education · Resident education · Virtual reality simulator · OSATS · VASAlthough the number of live operations in residents' training has diminished [1], technology offers new solutions for surgical training in the form of lap trainers and virtual reality simulators [2][3][4][5]. With these training tools, it is possible to acquire fundamental laparoscopic skills [2], and they may be used as a part of the procedural training curriculum [6], as well. Though procedural skills have been proven to be transferred into the operating room after virtual reality simulator modules in laparoscopic cholecystectomy [7], in laparoscopic salpingectomy [8], and in cataract surgery [9], no data exists on the effect of virtual reality simulators on advanced major surgical procedures [10].Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

show abstract

“…Data collection was performed in a controlled and standardised environment using the same investigator who also acted as the nurse in all procedures. Furthermore, meticulous rater training was performed to improve reliability (21,22).…”

Section: Validity Of the Assessment Toolmentioning

confidence: 99%

Assessment of competence in local anaesthetic thoracoscopy: development and validity investigation of a new assessment tool

Nayahangan¹,

Svendsen²,

Bødtger³

et al. 2021

J Thorac Dis

View full text Add to dashboard Cite

Background: The aims of the study were to develop an assessment tool in local anaesthetic thoracoscopy (LAT), investigate validity evidence, and establish a pass/fail standard.Methods: Validity evidence for the assessment tool was gathered using the unified Messick framework.The tool was developed by five experts in respiratory medicine and medical education. Doctors with varying experience performed two consecutive procedures in a standardized, simulation-based setting using a newly developed thorax/lung silicone model. Performances were video-recorded and assessed by four expert raters using the new tool. Contrasting groups' method was used to set a pass/fail standard.Results: Nine novices and 8 experienced participants were included, generating 34 recorded performances and 136 expert assessments. The tool had a high internal consistency (Cronbach's alpha =0.94) and high inter-rater reliability (Cronbach's alpha =0.91). The total item score significantly correlated with the global score (r s =0.86, P<0.001). Participants' first performance correlated to second performance (test-retest reliability) with a Pearson's r of 0.93, P<0.001. Generalisability (G) study showed a G-coefficient of 0.92 and decision (D) study estimated that one performance assessed by two raters or four performances assessed by one rater are needed to reach an acceptable reliability, i.e., G-coefficient >0.80. The tool was able to discriminate between the two groups in both performances: experienced mean score =30.8±4.2; novice mean score =15.8±2.3, P<0.001. Pass/fail standard was set at 22 points. Conclusions:The newly developed assessment tool showed solid evidence of validity and can be used to ensure competence in LAT.

show abstract

Effect of rater training on the reliability of technical skill assessments: a randomized controlled trial

Cited by 13 publications

References 34 publications

Development of a median sternotomy simulation model for cardiac surgery training

Development of a median sternotomy simulation model for cardiac surgery training

Simulator training and residents’ first laparoscopic hysterectomy: a randomized controlled trial

Assessment of competence in local anaesthetic thoracoscopy: development and validity investigation of a new assessment tool

Contact Info

Product

Resources

About