Inter-rater Reliability of a Clinical Documentation Rubric Within Pharmacotherapy Problem-Based Learning Courses

Abstract

Objective. To evaluate a clinical documentation rubric for pharmacotherapy problem-based learning (PBL) courses using inter-rater reliability (IRR) among different evaluators. Methods. A rubric was adapted for use in grading student pharmacists' clinical documentation in pharmacotherapy PBL courses. Multiple faculty evaluators used the rubric to assess student pharmacists' clinical documentation. The mean rubric score given by the evaluators and the standard deviation were calculated. Intra-class correlation coefficients (ICC) were calculated to determine the inter-rater reliability (IRR) of the rubric. Results. Three hundred seventeen clinical documentation submissions were scored twice by multiple evaluators using the rubric. The mean initial evaluation score was 9.1 (SD50.9) and the mean second evaluation score was 9.1 (SD50.9), with no significant difference found between the two. The overall ICC was 0.7 across multiple graders, indicating good IRR. Conclusion.The clinical documentation rubric demonstrated overall good IRR between multiple evaluators when used in pharmacotherapy PBL courses. The rubric will undergo additional evaluation and continuous quality improvement to ensure that student pharmacists are provided with the formative feedback they need.

Determination of Interrater Reliability of a Universal Evaluator Rubric to Assess Student Pharmacist Communication Skills

“…Previously published studies have described methods for testing and achieving interrater reliability, however these were limited to single evaluator types. [11][12] There is no rubric for assessment of communication skills among student pharmacists which has been validated for use among multiple evaluator types (ie, senior student colleagues, residents, faculty, preceptors, and/or standardized patients (SPs)).…”

Feedback for Learning in Pharmacy Education: A Scoping Review

“…These four papers tested the use of feedback in three different areas: improving CPR skills development [14]; subjective, objective, assessment, and plan (SOAP) note writing [15]; and student performance on verbal competency and patient counseling and interviewing assessments [16,17]. The other studies' feedback focus areas were patient communication including counseling and other simulated patient interactions [18][19][20][21][22]; patient work up including SOAP notes [18,[23][24][25][26][27][28] or oral case presentations and care plans [26,[29][30][31]; and other topics such as evidence appraisal and research skills [32,33]; self-assessments [34]; disease information [35]; pharmacy calculations [36]; and pharmacology experiments [37]. Eleven of the 12 major theory talk articles assessed the impact of feedback on student learning.…”