Extending Applications of Generalizability Theory-Based Bifactor Model Designs

Abstract

In recent years, researchers have described how to analyze generalizability theory (GT) based univariate, multivariate, and bifactor designs using structural equation models. However, within GT studies of bifactor models, variance components have been limited to those reflecting relative differences in scores for norm-referencing purposes, with only limited guidance provided for estimating key indices when making changes to measurement procedures. In this article, we demonstrate how to derive variance components for multi-facet GT-based bifactor model designs that represent both relative and absolute differences in scores for norm- or criterion-referencing purposes using scores from selected scales within the recently expanded form of the Big Five Inventory (BFI-2). We further develop and apply prophecy formulas for determining how changes in numbers of items, numbers of occasions, and universes of generalization affect a wide variety of indices instrumental in determining the best ways to change measurement procedures for specific purposes. These indices include coefficients representing score generalizability and dependability; scale viability and added value; and proportions of observed score variance attributable to general factor effects, group factor effects, and individual sources of measurement error. To enable readers to apply these techniques, we provide detailed formulas, code in R, and sample data for conducting all demonstrated analyses within this article.

Applying multivariate generalizability theory to psychological assessments.

“…Although we emphasize derivation of score consistency indices for composite scores using multivariate GT here, such indices also can be obtained using alternative GT methods. These alternatives include (a) analyzing item scores within a bifactor GT framework in which universe score variance is partitioned into two sources: general (common explained variance across all items) and group (unrelated explained variance unique to each subscale; see, e.g., Vispoel & Lee, 2023; Vispoel, Lee, Chen, & Hong, 2023b, 2023c; Vispoel et al, 2022, 2023), (b) replacing items with parallel splits balanced for subscale representation, item phrasing, and overall statistical characteristics of items within a univariate GT persons × splits × occasions design (see Vispoel et al, 2018a, 2018b, 2018c, 2018d; Vispoel, Xu, & Schneider, 2022b), and (c) ignoring division of items into subscales altogether and analyzing all items from the composite as a single domain using univariate GT (see Vispoel, Lee, Chen, & Hong, 2023c; Vispoel et al, 2022, 2023).…”

“…This is accomplished by replacing person variance in the numerator of Equation 8 with the index for the targeted source of measurement error as shown in Equations 11–13. Similar substitutions can be made for estimating proportions of absolute error for D coefficients (see, e.g., Vispoel, Lee, Chen, & Hong, 2023b; Vispoel & Tao, 2013): …”

Supplemental Material for Applying Multivariate Generalizability Theory to Psychological Assessments

“…This is accomplished by replacing person variance in the numerator of Equation 8 with the index for the targeted source of measurement error as shown in Equations 11-13. Similar substitutions can be made for estimating proportions of absolute error for D coefficients (see, e.g., Vispoel, Lee, Chen, & Hong, 2023b;Vispoel & Tao, 2013):…”

“…Although we emphasize derivation of score consistency indices for composite scores using multivariate GT here, such indices also can be obtained using alternative GT methods. These alternatives include (a) analyzing item scores within a bifactor GT framework in which universe score variance is partitioned into two sources: general (common explained variance across all items) and group (unrelated explained variance unique to each subscale; see, e.g., Vispoel, Lee, Chen, & Hong, 2023b, 2023cVispoel et al, 2022, (b) replacing items with parallel splits balanced for subscale representation, item phrasing, and overall statistical characteristics of items within a univariate GT persons × splits × design (see Vispoel et al, 2018aVispoel et al, , 2018cVispoel et al, , 2018dVispoel, Xu, & Schneider, 2022b), and (c) ignoring division of items into subscales altogether and analyzing all items from the composite as a single domain using univariate GT (see Vispoel, Lee, Chen, & Hong, 2023c;Vispoel et al, 2022.…”

A Robust Indicator Mean-Based Method for Estimating Generalizability Theory Absolute Error and Related Dependability Indices within Structural Equation Modeling Frameworks

“…Using SEMs to conduct GT analyses has many advantages including use of alternative estimation procedures to correct for scale coarseness effects (diagonally weighted least squares, paired maximum likelihood, etc. ; [14,38,42,[44][45][46]49]), derivation of Monte Carlo confidence intervals for key indices of interest [14,44,46,47,50,51,55,56], partitioning of variance at both total score and individual item levels [46][47][48][49], and extensions to multivariate [37,46,47,50,51] and bifactor model GT designs [46,50,[52][53][54]. These advantages stem in part from the inherent capabilities of SEM programs to tailor factor loadings, variances, residuals, intercepts, and thresholds to specific needs and contexts of assessment.…”