Assessing Within-Group Agreement

Abstract

The measure of within-group agreement most frequently encountered in organizational psychology is the r WG index. The r WG index is determined by comparing the observed group variance among raters with an expected random variance. The most critical issue in calculating the r WG is the choice of an appropriate random distribution that would be expected to follow from raters making their ratings at random. A data-driven approach that uses randomgroup resampling (RGR) procedures to determine the expected random variance has been proposed. In the present study, the application of the RGR procedure will be illustrated with reference to students' ratings of their mathematics instruction and critically compared with a recently proposed simulation-based approach. It will be shown mathematically that the probability of obtaining statistically significant within-group agreement when applying the RGR procedure strongly depends on the intraclass correlation as well as on the group sizes. Finally, implications for applying the RGR procedure to assess within-group agreement in multilevel data will be discussed.

Workplace racial/ethnic similarity, job satisfaction, and lumbar back health among warehouse workers: Asymmetric reactions across racial/ethnic groups

“…For job control, the ICC(1) was 0.07 (F(68, 308) = 1.39, p < .05), which indicates a small effect. The r wg(j) , a measure for withingroup agreement that compares the observed group variance among raters with an expected random variance (James, Demaree, & Wolf, 1984;Lüdtke & Robitzsch, 2009), was at an adequate level for both scales (the mean r wg(j) = .86 for quantitative workload and .71 for job control). All coefficients indicated that it is acceptable to aggregate perceived workload and job control at the job level.…”

Transformational leadership and job satisfaction in vocational high schools in Türkiye: a multilevel mediation model of teacher professional learning and self-efficacy

“…The reliability of the school-level means was evaluated by examining the ICC(2) for each measure, which varied from 0.41 to 0.82 and indicated strong indices (Shrout and Fleiss, 1979). Furthermore, all variables' Rwg values exceeded 0.70 (Lüdtke and Robitzsch, 2009) (See Table 2).…”

Enhancing Patient Physician Communication with Electronic Symptom Reporting (ESR): A Multilevel Model

“…Turel and Zhang, 2011). The ability to demonstrate that the aggregated values exhibit an acceptable degree of within-group agreement [James, Demaree and Wolf, 1984;Luedtke and Robitzsch, 2009] shows that this collective concept does indeed exist. In the clinical context, it is justified by the fact that physicians have their own style of communicating with patients [Reise and Duan, 1999] ; therefore, groups of patients can be homogenous with respect to disease and treatment-related concepts.…”