Improving the measurement of group-level constructs by optimizing between-group differentiation.

Abstract

The ability to detect differences between groups partially impacts how useful a group-level variable will be for subsequent analyses. Direct consensus and referent-shift consensus group-level constructs are often measured by aggregating group member responses to multi-item scales. We show that current measurement validation practice for these group-level constructs may not be optimized with respect to differentiating groups. More specifically, a 10-year review of multilevel articles in top journals reveals that multilevel measurement validation primarily relies on procedures designed for individual-level constructs. These procedures likely miss important information about how well each specific scale item differentiates between groups. We propose that group-level measurement validation be augmented with information about each scale item's ability to differentiate groups. Using previously published datasets, we demonstrate how ICC(1) estimates for each item of a scale provide unique information and can produce group-level scales with higher ICC(1) values that enhance predictive validity. We recommend that researchers supplement conventional measurement validation information with information about item-level ICC(1) values when developing or modifying scales to assess group-level constructs. (PsycINFO Database Record

Validation of the Copenhagen Psychosocial Questionnaire Version III and Establishment of Benchmarks for Psychosocial Risk Management in Sweden

“…In relation to the previous Swedish version, the present third version includes five new dimensions and six dimensions have changed name, one dimension has changed response options, 16 dimensions have a reduced number of items, two items are replaced and five have changes in wording. Decisions regarding the selection of dimensions were guided by the perceived relevance to the Swedish context, cognitive interviews, pilot tests and dialogue with stakeholders, taking the item level in the international COPSOQ III and item-level ICC(1) values into consideration for not jeopardizing the ability to differentiate workplaces, as recently suggested by Bliese and colleagues [35].…”

“…ICC(1) and ICC(2) were calculated for each dimension based on aggregation of individual level data to ISCO major occupational group (national sample) and to workplace (workplace sample). ICC(1) represents the amount of variance in the employees' responses that can be explained by their membership of a group (occupation or workplace) [35,[40][41][42]. ICC(1) values of 0.05 can be considered as a small to medium effect and higher values indicate stronger effects [42], ICC(2) is an estimate of reliability of the aggregated group means [35,40,41].…”

“…ICC(1) represents the amount of variance in the employees' responses that can be explained by their membership of a group (occupation or workplace) [35,[40][41][42]. ICC(1) values of 0.05 can be considered as a small to medium effect and higher values indicate stronger effects [42], ICC(2) is an estimate of reliability of the aggregated group means [35,40,41]. Values <0.5 indicate poor reliability, 0.5-0.75 moderate and >0.75 indicate good reliability of group-level means [43].…”

Benchmarks for Evidence-Based Risk Assessment with the Swedish Version of the 4-Item Psychosocial Safety Climate Scale

“…ICC(1) and ICC(2) were calculated. ICC(1) represents the amount of variance in the employees' responses that can be explained by their membership of a group (workplace) [42][43][44][45]. ICC(1) values of 0.05 can be considered as a small to medium effect, and higher values indicate stronger effects, i.e., a larger proportion of the variance explained by the workplace [45].…”

“…ICC(1) values from applied field research of organizations is typically up to a maximum of 0.20 (p 362 in [43]). ICC(2) is an estimate of reliability of the aggregated group means [42][43][44]. Values <0.5 indicate poor reliability, 0.5-0.75 is moderate, and >0.75 indicates good reliability of group-level means [46].…”