Selecting Null Distributions When Calculating r_wg

Abstract

rwg is a common metric used to quantify interrater agreement in the organizational sciences. Finn developed rwg but based it on the assumption that raters’ deviations from their true perceptions are influenced by random chance only. James, Demaree, and Wolf extended Finn’s work by describing procedures to account for the additional influence of response biases. We demonstrate that organizational scientists have relied largely on Finn’s procedures, at least in part because of a lack of specific guidance regarding the conditions under which various response biases might be present. In an effort to address this gap in the literature, we introduce the concept of target-irrelevant, nonrandom forces (those aspects of the research context that are likely to lead to response biases), then describe how the familiar “5Ws and an H” framework (i.e., who, what, when, where, why, and how) can be used to identify these biases a priori. It is our hope that this system will permit those who calculate rwg to account for the effects of response biases in a manner that is simultaneously rigorous, consistent, and transparent.

Self-reliance: A Gender Perspective on its Relationship to Communality and Leadership Evaluations

“…At high levels of self-reliance (1 SD above the mean) this pattern reversed, B 5 20.08, SE 5 .09, p 5 .34. 4 1 We specified a slightly skewed distribution to account for the raters' presumed leniency bias (Meyer, Mumford, Burrus, Campion, & James, 2014). We had reason to believe that because of the raters' familiarity and personal relationships with the ratees, the deviations from raters' true perceptions in our sample were not caused solely by random, nontarget-specific factors (Meyer et al, 2014).…”

“…4 1 We specified a slightly skewed distribution to account for the raters' presumed leniency bias (Meyer, Mumford, Burrus, Campion, & James, 2014). We had reason to believe that because of the raters' familiarity and personal relationships with the ratees, the deviations from raters' true perceptions in our sample were not caused solely by random, nontarget-specific factors (Meyer et al, 2014). 2 Triandis & Gelfand (1998) assessed the relationship between horizontal individualism and 75 items that measured self-reliance, competition, emotional distance from in-groups, hedonism, family integrity, interdependence, and sociability to determine which of these different clusters best predicted horizontal individualism and the other scales.…”

How Knowledge Worker Teams Deal Effectively with Task Uncertainty: The Impact of Transformational Leadership and Group Development

“…While Biemann et al (2012) recommend not deleting groups with low agreement, in favor of test power, we considered deleting such teams and thus sacrificing test power as the more conservative approach. Despite the known criticism (LeBreton and Senter, 2008), in this case using an equal distribution was justified by three reasons: (1), we only used r wg(j) for comparisons among teams, which means that any bias introduced by a potentially inadequate null distribution would affect all teams equally; (2) none of the restrictions mentioned by Meyer et al (2014) seemed applicable to our data and thus no other distribution was more favorable, and (3) the null distribution was frequently used in recent leadership research (Biemann et al, 2012), which increases the comparability among studies. We deleted 26 teams in which either one r wg(j) value was below 0.40, or in which four r wg(j) values were below 0.70.…”

Human resource practices and migrant workers' turnover intentions: The roles of post‐migration place identity and justice perceptions

“…The aggregation statistics were acceptable. Under the most commonly used rectangular distribution (Meyer, Mumford, Burrus, Campion, & James, ), the median r wg was 0.98 and the mean r wg was 0.95 ( SD = 0.10). Additionally, we conducted a simulation‐based statistical significance test to ensure that the senior managers exhibited greater‐than‐chance agreement in terms of HR evaluations (Smith‐Crowe, Burke, Cohen, & Doveh, ).…”