Temporal and group-related trends in peer assessment amongst medical students

Abstract

Peer assessment has been increasingly recommended as a way to evaluate the professional competencies of medical trainees. Prior studies have only assessed single groups measured at a single timepoint. Thus, neither the longitudinal stability of such ratings nor differences between groups using the same peer-assessment instrument have been reported previously. Participants were all members of 2 consecutive classes of medical students (n = 77 and n = 85) at the University of Rochester School of Medicine and Dentistry who completed Years 2 and 3 of medical school consecutively. All participants were evaluated by 6-12 classmates near the end of both Years 2 and 3. Main outcome measures were mean numerical ratings on peer-assessed scales of professional work habits (WH) and interpersonal attributes (IA). Both scales had high internal consistencies in both years (Cronbach's alpha 0.84-0.94). The IA and WH scales were moderately correlated with one another (r = 0.36 in Year 2, r = 0.28 in Year 3). Year 2 scores were predictive of Year 3 scores for both scales (WH: r = 0.64; IA; r = 0.62). Generalisability and decision analyses revealed that 1 class was consistently more discriminating with the WH scale, while the other was more discriminating with of the IA scale. Depending on the class, year and scale, the number of raters needed to achieve a reasonable reliability ranged between 7 and 28. Although Year 3 peer ratings were consistently higher than Year 2 peer ratings for both WH and IA, individual scores were highly correlated across the 2 years, despite the fact that different individuals were chosen as peer raters. Abilities appear to be stable between Years 2 and 3 of medical school. Groups may differ in their ability to discriminate different kinds of skills. Generalisability analysis can be used to discover these patterns within groups.

Peer Assessment of Professionalism: A Five-Year Experience in Medical Clerkship

“…Many of the extant programs request learners to report on a limited number of peers (either randomly selected or self selected). 6 Trainees often dislike and resist peer evaluations, stating that they hinder team relationships, promote competitiveness, and may cause harm by leading to an unfavorable grade. [7][8][9] Students generally agree that a peer assessment system should: (1) be 100% anonymous, (2) provide immediate feedback, (3) focus on both unprofessional and professional behaviors, and (4) be used formatively to reward exemplary behavior and address repetitive professional lapses.…”

Reliability and validity of student peer assessment in medical education: A systematic review

“…Using the extremes 'poor' and 'good' only (omitting the middle classification), these percentages were, respectively, 87% (only progress test predictor significant) and 87% (OSCE only) Kovach et al (2009) Two domains (knowledge, interpersonal relationships) Predictive validity: peer knowledge ratings were highly predictive for performance on the NBME, whereas peer ratings on the relationship factor were not statistically significant as predictors. In the case of final grades in surgery, both peer ratings on knowledge and relationships were highly related, thus, good predictors Lurie et al (2006a) Second year medical students over a period of 2 years (two consecutive classes: Each student evaluated all peers in group.…”

“…In total, the 28 included studies described 22 different instruments for peer assessment in mainly medical educational settings. Three studies by Lurie et al (2006aLurie et al ( ,b, 2007 were based on the Peer Assessment Protocol by Dannefer et al (2005), Arnold et al (1981) included the clinical Performance Evaluation by Willoughby et al (1979), Papinckzak et al (2007a,b) published two studies using the same questionnaire, and Linn et al (1975Linn et al ( , 1976 presented two different versions of the same peer assessment questionnaire.…”

“…This content validity was usually explained referring to existing literature or frameworks. Criterion validity, frequently mentioned in Table 2, refers to the relationship linking the attributes in a tool with the performance on a criterion, whereas predictive validity (Linn et al 1976;Arnold et al 1981;Dijcks et al 2003;Lurie et al 2007) indicates the degree to which test scores predict performance on some future criterion and convergent validity (Lurie et al 2006a) the extent to which different measures of the same construct correlate with one other (DeVon et al 2007). Although no single criterion or gold standard will ever be perfect (Norman et al 1996), usually, faculty ratings are considered to be the gold standard in educational settings (Lin et al 1975;van Rosendaal & Jennett 1994;Davis 2002;Dijcks et al 2003;Bryan et al 2005;Roark et al 2006).…”

Assessment in Medical Education; What Are We Trying to Achieve?

“…Otherwise, it has potential to be undermining, destructive and divisive (Lurie SJ, Nofziger AC, Meldrum S et al, 2006). In addition, peer assessment has been shown to be consistent from year to year regardless of the way assessors are selected (Lurie S, Lambert D, Grady Weliky TA, 2007).…”