Can student self-ratings be compared with peer ratings? A study of measurement invariance of multisource feedback

Abstract

Measurement invariance is a prerequisite for comparing measurement scores from different groups. In medical education, multi-source feedback (MSF) is utilized to assess core competencies, including the professionalism. However, little attention has been paid to the measurement invariance of assessment instruments; that is, whether an instrument holds the same meaning across different rater groups. To examine the measurement invariance of the National Taiwan University professionalism MSF (NTU P-MSF) in order to determine whether medical students' self-rating can be compared to their peers' rating. An eight-factor model was specified for confirmatory factor analysis to examine the construct validity of the NTU P-MSF. Cronbach's alpha was computed for the items of each domain to evaluate internal consistent reliability. The same eight-factor model was used for multi-group confirmatory factor analyses. Four hierarchical models were specified to test configural (i.e., identical factor-item relationship), metric (i.e., identical factor loadings), scalar (i.e., identical intercepts), and error variance across self-rating and peer rating groups. One hundred and twenty second-year medical students from weekly discussion groups conducted as part of a medical professionalism course agreed to use the NTU P-MSF to assess themselves or their discussion group peers. NTU P-MSF assessment scores were a good fit for the eight-factor model among self group and peer group. The Cronbach's alpha coefficients of students' NTU P-MSF scores and peers' scores ranged from 0.76 to 0.89 and 0.84 to 0.91, respectively indicating that the NTU P-MSF scores also have good internal consistent reliability between both groups. In addition, same factor structure and similar factor loadings and intercepts of NTU P-MSF scores between both groups indicate that NTU P-MSF scores had configural, metric, and scalar invariance. Thus, students' self-assessments and peer assessments can be compared in terms of the constructs of NTU P-MSF scores, change in NTU P-MSF scores, and its factor scores. This study demonstrates how to investigate the measurement invariance of a professionalism MSF and contributes to the discussion on self- and peer assessment in medical education.

Objectivity in subjectivity: do students’ self and peer assessments correlate with examiners’ subjective and objective assessment in clinical skills? A prospective study

“…Like others have reported, 25 SA and PA correlated with each other generally as well as at the level of their subdomains in our study also. Interestingly, both SA and PA ratings are much higher than ESA.…”

“…This continuity can be included into students’ portfolios; in a similar manner to ‘multisource feedback’ discussed by others. 25 This study incites fresh excitation in an old discussion. Simplified subjective tools need repeated adaptation, validation, reliability check and evaluation based on the needs and settings of a course, and students’ level.…”

The Objective Structured Clinical Examination as an Assessment Strategy for Clinical Competence in Novice Nursing Practitioners in Taiwan

“…In addition, the OSCE helped increase the NNPs' con dence and reduce their personal embarrassment when encountering similar clinical situations. Previous reports showed that nursing students found that the OSCE help them deal with stressful clinical situations and develop their con dence in clinical practice [24][25][26][27]. Student midwives saw the OSCE as a valid means of assessment and that it increased their con dence in performing clinical skills [26,28].…”

Individuation Test for Emerging Adults (ITEA): Adaptação para a população portuguesa

“…Para além disso, este estudo estabelece evidências importantes sobre a invariância do ITEA. Embora a invariância de medição seja um pré-requisito para a realização de comparações entre grupos, raramente é testada (Lee, Tsai, Chiu, & Ho, 2016). Assim, a falta de variação do ITEA entre os papéis parentais, bem como entre o género dos participantes, sugere uma evidência importante para a robustez dessa medida de avaliação do processo de individuação aos pais em jovens adultos.…”