Applying multivariate generalizability theory to psychological assessments.

Abstract

Multivariate generalizability theory (GT) represents a comprehensive framework for quantifying score consistency, separating multiple sources contributing to measurement error, correcting correlation coefficients for such error, assessing subscale viability, and determining the best ways to change measurement procedures at different levels of score aggregation. Despite such desirable attributes, multivariate GT has rarely been applied when measuring psychological constructs and far less often than univariate techniques that are subsumed within that framework. Our purpose in this tutorial is to describe multivariate GT in a simple way and illustrate how it expands and complements univariate procedures. We begin with a review of univariate GT designs and illustrate how such designs serve as subcomponents of corresponding multivariate designs.Our empirical examples focus primarily on subscale and composite scores for objectively scored measures, but guidelines are provided for applying the same techniques to subjectively scored performance and clinical assessments. We also compare multivariate GT indices of score consistency and measurement error to those obtained using alternative GT-based procedures and across different software packages for analyzing multivariate GT designs. Our online supplemental materials include instruction, code, and output for common multivariate GT designs analyzed using mGENOVA and the gtheory, glmmTMB, lavaan, and related packages in R.

"Determining When Subscale Scores from Assessment Measures Provide Added Value"

“…Haberman (2008)'s method is based on computation of indices for a subscale and its associated composite reflecting reduction in measurement-related error when estimating the underlying construct(s) represented by the subscale's scores. Technically, the indices for the subscale and composite reflect proportional reductions in mean-squared error (PRMSE, [4][5][6][7][8][9][10][11][12]) when estimating true scores from observed scores. These indices, in turn, can then be used to create a value-added ratio (VAR; see [7]) by dividing the PRMSE for the subscale by the PRMSE for the composite scores as shown in Equation (1).…”

“…A useful way to address this problem is to apply generalizability theory-based prophecy techniques [15][16][17][18][19] to determine the extent to which increases in numbers of items might improve subscale added value (see [8][9][10][11][12] for further details). The remaining columns (3)(4)(5)(6) in (Table 1) include estimates of VARs for each subscale when pairs of items are successively added up to a maximum of 12 total items.…”

“…To answer this question, measurement specialists have produced a variety of indices to quantify subscale viability. In this article, we illustrate one such procedure first described by Haberman (2008, [4], also see [5][6][7]) that can encompass a wide variety of mea-surement paradigms including classical test theory, generalizability theory, item response theory, and factor analytic techniques [8][9][10][11][12].…”

A Robust Indicator Mean-Based Method for Estimating Generalizability Theory Absolute Error and Related Dependability Indices within Structural Equation Modeling Frameworks

“…Such techniques have recently been applied to advantage in such diverse fields as education [5][6][7][8][9][10][11][12][13], psychology [14][15][16][17], business [18][19][20][21], medicine/health sciences [22][23][24][25][26][27][28][29], psychophysiology [30][31][32], athletic training [33][34][35], and many others. Although GT designs have traditionally been analyzed using analysis of variance (ANOVA) procedures, they also can be analyzed using linear mixed-effect [36,37] and structural equation models (SEMs; [14,[37][38][39][40][41][42][43][44][45][46][47][48][49][50]…”

“…Using SEMs to conduct GT analyses has many advantages including use of alternative estimation procedures to correct for scale coarseness effects (diagonally weighted least squares, paired maximum likelihood, etc. ; [14,38,42,[44][45][46]49]), derivation of Monte Carlo confidence intervals for key indices of interest [14,44,46,47,50,51,55,56], partitioning of variance at both total score and individual item levels [46][47][48][49], and extensions to multivariate [37,46,47,50,51] and bifactor model GT designs [46,50,[52][53][54]. These advantages stem in part from the inherent capabilities of SEM programs to tailor factor loadings, variances, residuals, intercepts, and thresholds to specific needs and contexts of assessment.…”