Inter-rater reliability and generalizability of patient note scores using a scoring rubric based on the USMLE Step-2 CS format

Abstract

Recent changes to the patient note (PN) format of the United States Medical Licensing Examination have challenged medical schools to improve the instruction and assessment of students taking the Step-2 clinical skills examination. The purpose of this study was to gather validity evidence regarding response process and internal structure, focusing on inter-rater reliability and generalizability, to determine whether a locally-developed PN scoring rubric and scoring guidelines could yield reproducible PN scores. A randomly selected subsample of historical data (post-encounter PN from 55 of 177 medical students) was rescored by six trained faculty raters in November-December 2014. Inter-rater reliability (% exact agreement and kappa) was calculated for five standardized patient cases administered in a local graduation competency examination. Generalizability studies were conducted to examine the overall reliability. Qualitative data were collected through surveys and a rater-debriefing meeting. The overall inter-rater reliability (weighted kappa) was .79 (Documentation = .63, Differential Diagnosis = .90, Justification = .48, and Workup = .54). The majority of score variance was due to case specificity (13 %) and case-task specificity (31 %), indicating differences in student performance by case and by case-task interactions. Variance associated with raters and its interactions were modest (<5 %). Raters felt that justification was the most difficult task to score and that having case and level-specific scoring guidelines during training was most helpful for calibration. The overall inter-rater reliability indicates high level of confidence in the consistency of note scores. Designs for scoring notes may optimize reliability by balancing the number of raters and cases.

Correlation between physician assistant students’ performance score of history taking and physical exam documentation and scores of Graduate Record Examination, clinical year grade point average, and score of Physician Assistant National Certifying Exam in the United States

“…Extensive training and education were provided by the PA program faculty. Preexisting grading rubrics were used by faculty to grade the H&Ps to promote inter-rater reliability [ 10 ] ( Supplement 1 ). Inter-rater reliability was assessed by dual grading of a series of H&P papers by 2 random graders and calculating an intraclass correlation coefficient (ICC).…”

Pre‐medical majors in the humanities and social sciences: impact on communication skills and specialty choice

“…The CIS component of the GCE is measured using a 4‐point behaviourally anchored rating scale composed of 14 items (see Table for a full list of these items), rated by the SP directly after each encounter . The PN is scored by trained physician raters using a scoring rubric that refers to four tasks; a prior study of this rubric showed high inter‐rater reliability (weighted κ = 0.79) and validity evidence supporting its use …”

“…24 The PN is scored by trained physician raters using a scoring rubric that refers to four tasks; a prior study of this rubric showed high interrater reliability (weighted j = 0.79) and validity evidence supporting its use. 26…”

Characteristics and Implications of Diagnostic Justification Scores Based on the New Patient Note Format of the USMLE Step 2 CS Exam

“…Once training and calibration were completed, raters scored their assigned notes individually online over a period of about one month, using CAE LearningSpace. 13 In a separate rater analysis, 14 raters. 15 The full dataset using single ratings per note was used for this study.…”