Interrater Reliability of an Oral Case Presentation Rating Tool in a Pediatric Clerkship

Abstract

The Patient Presentation Rating tool is a reliable instrument for evaluating medical students' oral patient presentations.

Feedback on Oral Presentations During Pediatric Clerkships: A Randomized Controlled Trial

“…Although both forms include the overall presentation quality scale, the detailed form also includes 17 items presented with the use of behavioral anchors designed to incorporate elements of clinical reasoning into the feedback. 10 Several limitations to this study warrant consideration. First, miscommunications among our study team resulted in 3 clerkship blocks not receiving their randomly assigned experience, which we conservatively addressed by reporting results of intention-to-treat analyses as our primary findings; therefore, our results may underestimate the impact of the interventions on presentation quality, as the Supplemental Information and Supplemental Tables 6 and 7 show.…”

“…Third, although outcome evaluators at all sites received similar training before the sessions, we did not perform cross-institutional checks for uniformity in outcome evaluation standards, so differing expectations of presentations at the 4 schools may have influenced outcome ratings. Fourth, although a 5-point Likert scale version of questions assessing overall presentation quality was shown to be reliable in rating taped presentations, 10 neither the reliability of the 9-point Likert scale version of this question nor its use in rating "live" presentations has been assessed. In addition, although the detailed form has face validity, it has not been formally validated, nor has the instrument used to assess student report of presentation improvement.…”

“…However, feedback on presentations given by session facilitators who completed a single-item evaluation form did not improve students' subsequent presentations. Although the "detailed" session facilitators completed a reliable 18-item instrument that has face validity, 10 the tool has not been formally validated, so the precise educational impact of the measured 3.8% higher faculty-rated presentation quality scores is unclear. Although this effect size is a real difference, some teachers may dismiss it as insufficient to justify adding OP feedback sessions to the clerkship.…”

“…First, miscommunications among our study team resulted in 3 clerkship blocks not receiving their randomly assigned experience, which we conservatively addressed by reporting results of intention-to-treat analyses as our primary findings; therefore, our results may underestimate the impact of the interventions on presentation quality, as the Supplemental Information and Supplemental Tables 6 and 7 show. Second, although the primary outcome was assessed with a reliable instrument 10 by evaluators oriented to its use, we did not record the feedback or the outcome assessment sessions; doing so could have confirmed the feedback provided during the detailed and simple sessions differed, and clarified what aspects of presentations improved during the study. However, because the only difference between simple and detailed sessions was the form faculty completed, our results are consistent with the feedback given during the sessions being different.…”

“…In the detailed sessions, facilitators evaluated presentations by using an 18-item form that assessed presentation quality overall on the same 9-point Likert scale as well as 17 presentation-specific domains (see Table 1) rated on 5-point Likert scales previously shown have a reliable intraclass correlation coefficient of 0.9. 10 Participants did not receive copies of either evaluation form.…”