Klinisch-praktische Prüfungen in der orthopädischen Lehre: Wer ist der „ideale“ Prüfer?

Abstract

Medical curricula require an increasing amount of integrated clinical practical skills education. German medical faculties are increasingly implementing the OSCE (objective structured clinical examination) to test students' practical as well as communication skills. Teaching and assessment tools for practical skills include simulators and simulated patient scenarios, where peer-assisted teaching (PAT) is applied. Trained student tutors support this process. This study examines for the first time the use of PAT student tutors as examiners in an orthopaedic OSCE unit and the existence of possible differences in rating between faculty and PAT student tutors. N = 94 students (out of n = 324) in their first clinical year were randomly assigned to the OSCE unit "spine and pelvis examination" in March/April 2016 and data were collected over the three-day OSCE period. A structured checklist was developed and the two examiners (one university professor from the Department of Orthopaedics, one PAT student tutor) received identical preparatory training. The checklist was completed by both examiners independently and data were collected for statistical analysis. The analysis focused on the overall evaluation in points, as well as separate checklist sections focusing on different competences and comparison of the three consecutive examination days. Analysis was conducted by SPSS. The average number of points in the overall evaluation was 19.5 (out of 25) points. The student tutor evaluated the students with an average of 19.1 points, the professor with 19.9 points. A significant difference was observed between the two examiner evaluations of day one (p < 0.001) and in the overall analysis (p < 0.001). On OSCE examination days 2 and 3, no significant differences in evaluation were observed. The analysis shows only one significant evaluation difference between the two assessors, with the PAT student tutor's evaluation being stricter. The authors conclude that a point difference of 0.8 out of 25 in the average overall evaluation and an interrater reliability of 0.95 in an orthopaedic OSCE station examining practical skills as well as communication skills justifies employing PAT student tutors as assessors in this context. The obligatory requirements for this are a detailed PAT student tutor training, the development of a well-structured and valid checklist, as well as continuous quality assurance of the overall OSCE.

OSABSS: An authentic examination for assessing basic surgical skills in surgical residents

“…In line with this, Kassam A and colleagues have suggested that implementing OSCEs in residency programs can aid in evaluating professional competencies [ 20 ]. Similar views are expressed by Melcher P et al, emphasizing the importance and implementation requirements of the OSCE exam in orthopedic residents [ 23 ]. In another study, Phillips D and colleagues developed an OSCE exam aligned with the milestones of the orthopedic education course and the six competencies approved by the ACGME [ 24 ].…”

“…They also stress the need for meticulous attention to the quality of exam administration [ 21 ]. In a similar vein, Melcher P, Roth A, and colleagues highlight similar issues, including evaluator training, standardized patients, appropriate scenario design, and checklists in evaluating exam quality [ 23 ]. The AMEE Guideline 81 provides detailed guidance on the process of scenario development, station preparation, evaluator training, and scoring and evaluation procedures.…”

Formative assessment of practical skills with peer-assessors: quality features of an OSCE in general medicine at the Heidelberg Medical Faculty

“…When comparing the student tutors with faculty members, the PAL students achieve the same results in final exams [ 8 ], [ 9 ], [ 10 ], [ 11 ] and the same or even higher quality of feedback [ 10 ]. Prerequisites for this are precisely defined student tutor training courses and checklists [ 12 ], [ 13 ].…”