Central tendency and matched difference approaches for assessing interrater agreement.

Abstract

In Study 1 of this two-part investigation, we present a "central tendency approach" and procedures for assessing overall interrater agreement across multiple groups. We define parameters for mean group agreement and construct bootstrapped confidence intervals around the mean population parameters for r, AD, and ICC(1). In Study 2, we extend assessments of overall interrater agreement by developing a "matched difference approach" and procedures for assessing real versus pseudo agreement in a sample of groups. Here, we use random group resampling and the matched difference between assessments of the respective r, AD, and ICC(1) values for actual and pseudo groups, with the establishment of bootstrapped confidence intervals around such differences. In both studies, we employ simulated and real data to demonstrate the accuracy and practical utility of the new procedures for assessing agreement with respect to groups. Notably, to generate simulated data for Studies 1 and 2, we developed a new underlying model for multilevel data and procedure for data generation, and we discuss its potential utility for enhancing research in group-level studies. Moreover, we discuss, relative to current practices, how and why the new inference procedures provide information about mean interrater agreement in the population, which can improve data aggregation decisions and interpretations of findings from group-level studies. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

How store managers’ employee climate perceptions affect frontline employee, customer and store performance outcomes: an examination in the small-store setting

“…Finally, empirical evidence shows that FLE perceived store manager support, FLE organizational commitment and customer satisfaction data exhibited adequate within-group agreement to justify aggregation to the store level. As we are applying a direct consensus model for these variables, the threshold for aggregation is met by evidence of high levels of within-group interrater agreement (Burke et al , 2018). Following the approach used by Schneider et al (1998), we computed r wg(j) statistics for FLE perceived store manager support, FLE organizational commitment and customer satisfaction from each store.…”

Teams in the wild are not extinct, but challenging to research: A guide for conducting impactful team field research with 10 recommendations and 10 best practices

“…With the ever-changing demands for what constitutes quality research due to methodological advancements (e.g., Burke et al, 2018;Meuer & Rupietta, 2017) and fluctuating workforce patterns (e.g., Schwartz et al, 2016), we felt it necessary to limit our review to studies published within the last 5 years; this ensures the findings communicated align with current research and not outdated trends. As such, we reviewed papers published between the years of 2014 and 2018 on teams using field data in leading journals from the disciplines of IO/OB (i.e., Academy of Management Journal, Journal of Management, Journal of Applied Psychology, Journal of Organizational Behavior, and Personnel Psychology).…”

Data aggregation adequacy testing in multilevel research: A critical literature review and preliminary solutions to key issues

“…评定项目只有 1 个时记作 r WG (1) , 有 J 个平行项目时记作 (Biemann et al, 2012;Castro, 2002;James et al, 1984;Kozlowski & Hattrup, 1992; (Carron et al, 2003;; 也有学者建议进行敏感性分析, 即 在剔除和不剔除两种情况下分别分析数据, 比较 结果是否有显著差异 (Biemann et al, 2012;Woehr et al, 2015); 还有学者明确指出不合格的组不应 保留, 否则会导致构念间的效应缺失(missed)、虚 假效应(misidentified)或错误解释(misinterpreted) (Castro, 2002;Van Mierlo et al, 2009)。 从实践来看, 国外大多数研究都选择保留所有样本组 (Burke et al, 2018)…”