Within-Group Agreement Scores: Using Resampling Procedures to Estimate Expected Variance.

Bliese, Paul D.; Halverson, Ronald R.; Rothberg, J.

doi:10.5465/ambpp.1994.10344875

Cited by 44 publications

(63 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The extra 49 groups reflect the fact that the number of randomly formed groups must be a product of the number of actual groups, since random groups are created to match size characteristics of actual groups (Bliese et al, 1995). (Bliese et al, 1994(Bliese et al, , 1995. Routine compared actual group variances to 5,049 randomly formed group variances (see Footnote 3).…”

Section: Agreement Score Resultsmentioning

confidence: 99%

“…Results from randomly formed groups are valuable in group-level analyses because they allow one to determine whether group results are due to the process of aggregation or due to some actual group phenomenon. To date, random group creation has been applied to the problem of estimating expected variances in within-group agreement scores (Bliese et al, 1994(Bliese et al, , 1995 and the interpretation of equivocal WABA results in this paper. We believe that examining results from randomly formed groups helps to identify group effects and to interpret group-level results.…”

Section: Equivocal Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Individual and Nomothetic Models of Job Stress: An Examination of Work Hours, Cohesion, and Well‐Being¹

Bliese

Halverson

1996

J Applied Social Pyschol

145

135

View full text Add to dashboard Cite

Many models of job stress are implicitly based on the assumption that there is considerable variability in how individuals perceive and respond to their environments. In this paper, we introduce a nomothetic perspective of job stress. The nomothetic perspective assumes that despite individual differences there will be consistencies in how groups of individuals perceive and respond to similar work environments. To contrast the individual and nomothetic perspectives, we analyzed data from 7,382 respondents from 99 groups. In the analyses, we examined individual‐and group‐level relationships using both real groups and randomly formed groups. The results revealed that respondents from the same work group agreed about perceptions of the work climate. The results also revealed that both individual and nomothetic perspectives were useful in describing the relationship between cohesion and psychological well‐being. The relationship between work hours and psychological well‐being, however, was best modeled from a nomothetic perspective.

show abstract

Section: Agreement Score Resultsmentioning

confidence: 99%

Section: Equivocal Resultsmentioning

confidence: 99%

Individual and Nomothetic Models of Job Stress: An Examination of Work Hours, Cohesion, and Well‐Being¹

Bliese

Halverson

1996

J Applied Social Pyschol

145

135

View full text Add to dashboard Cite

show abstract

“…To circumvent this problem, Bliese (2000;Bliese et al, 1994) proposed an RGR procedure that determines the variance within groups that would be expected if individuals made their ratings at random and agreement were attributable to response bias only. The present study extended Bliese's implementation of the RGR procedure (Bliese, 2006) in two critical respects.…”

Section: Discussionmentioning

confidence: 99%

“…Hence, with increasing numbers of actual groups, even small differences between the averaged observed group variance and average pseudo group variance will become statistically significant. Based on these results, the dependence of the RGR procedure on the ICC can be further illustrated by calculating an average r WG(RGR) index that is based on the average observed within-group variance S 2 • , using the variance S 2 ••, * of the empirical distribution of the pseudo class variances as a benchmark (Bliese et al, 1994):…”

Section: Homoscedasticity Of Within-group Variancesmentioning

confidence: 99%

Assessing Within-Group Agreement

Lüdtke

Robitzsch

2008

Organizational Research Methods

View full text Add to dashboard Cite

The measure of within-group agreement most frequently encountered in organizational psychology is the r WG index. The r WG index is determined by comparing the observed group variance among raters with an expected random variance. The most critical issue in calculating the r WG is the choice of an appropriate random distribution that would be expected to follow from raters making their ratings at random. A data-driven approach that uses randomgroup resampling (RGR) procedures to determine the expected random variance has been proposed. In the present study, the application of the RGR procedure will be illustrated with reference to students' ratings of their mathematics instruction and critically compared with a recently proposed simulation-based approach. It will be shown mathematically that the probability of obtaining statistically significant within-group agreement when applying the RGR procedure strongly depends on the intraclass correlation as well as on the group sizes. Finally, implications for applying the RGR procedure to assess within-group agreement in multilevel data will be discussed.

show abstract

“…The objective is to create a sampling distribution for the IRA statistic with an expected mean and standard deviation, which can be used to generate confidence intervals and significance tests. Random group resampling involves constructing a sampling distribution by repeatedly sampling and forming random groups from observations in the observed data set, and comparing the significance of the mean difference in within-group variances of the observed distribution and the randomly generated distribution using a Z -test (Bliese et al, 2000; Bliese and Halverson, 2002; Ludtke and Robitzsch, 2009), for which commands are available in R . Thus, significance testing of the S wg is possible through the random-group resampling approach.…”

Section: Standards For Agreementmentioning

confidence: 99%

An Overview of Interrater Agreement on Likert Scales for Researchers and Practitioners

O’Neill

2017

Front. Psychol.

View full text Add to dashboard Cite

Applications of interrater agreement (IRA) statistics for Likert scales are plentiful in research and practice. IRA may be implicated in job analysis, performance appraisal, panel interviews, and any other approach to gathering systematic observations. Any rating system involving subject-matter experts can also benefit from IRA as a measure of consensus. Further, IRA is fundamental to aggregation in multilevel research, which is becoming increasingly common in order to address nesting. Although, several technical descriptions of a few specific IRA statistics exist, this paper aims to provide a tractable orientation to common IRA indices to support application. The introductory overview is written with the intent of facilitating contrasts among IRA statistics by critically reviewing equations, interpretations, strengths, and weaknesses. Statistics considered include rwg, rwg*, r′wg, rwg(p), average deviation (AD), awg, standard deviation (Swg), and the coefficient of variation (CVwg). Equations support quick calculation and contrasting of different agreement indices. The article also includes a “quick reference” table and three figures in order to help readers identify how IRA statistics differ and how interpretations of IRA will depend strongly on the statistic employed. A brief consideration of recommended practices involving statistical and practical cutoff standards is presented, and conclusions are offered in light of the current literature.

show abstract

Within-Group Agreement Scores: Using Resampling Procedures to Estimate Expected Variance.

Cited by 44 publications

References 0 publications

Individual and Nomothetic Models of Job Stress: An Examination of Work Hours, Cohesion, and Well‐Being¹

Individual and Nomothetic Models of Job Stress: An Examination of Work Hours, Cohesion, and Well‐Being¹

Assessing Within-Group Agreement

An Overview of Interrater Agreement on Likert Scales for Researchers and Practitioners

Contact Info

Product

Resources

About

Within-Group Agreement Scores: Using Resampling Procedures to Estimate Expected Variance.

Cited by 44 publications

References 0 publications

Individual and Nomothetic Models of Job Stress: An Examination of Work Hours, Cohesion, and Well‐Being1

Individual and Nomothetic Models of Job Stress: An Examination of Work Hours, Cohesion, and Well‐Being1

Assessing Within-Group Agreement

An Overview of Interrater Agreement on Likert Scales for Researchers and Practitioners

Contact Info

Product

Resources

About

Individual and Nomothetic Models of Job Stress: An Examination of Work Hours, Cohesion, and Well‐Being¹

Individual and Nomothetic Models of Job Stress: An Examination of Work Hours, Cohesion, and Well‐Being¹