A Robust Indicator Mean-Based Method for Estimating Generalizability Theory Absolute Error and Related Dependability Indices within Structural Equation Modeling Frameworks

Abstract

In this study, we introduce a novel and robust approach for computing Generalizability Theory (GT) absolute error and related dependability indices using indicator intercepts that represent observed means within structural equation models (SEMs). We demonstrate the applicability of our method using one-, two-, and three-facet designs with self-report measures having varying numbers of scale points. Results for the indicator mean-based method align well with those obtained from the GENOVA and R gtheory packages for doing conventional GT analyses and improve upon previously suggested methods for deriving absolute error and corresponding dependability indices from SEMs when analyzing three-facet designs. We further extend our approach to derive Monte Carlo confidence intervals for all key indices and to incorporate estimation procedures that correct for scale coarseness effects commonly observed when analyzing binary or ordinal data.

Multivariate Structural Equation Modeling Techniques for Estimating Reliability, Measurement Error, and Subscale Viability When Using Both Composite and Subscale Scores in Practice

“…These extensions within G-theory designs are manifested in the derivation of variance components from random effects analysis of variance (ANOVA) models to produce G coefficients that reflect the extent to which results can be generalized to the broader domains represented by the measurement facets and that subsequently can be used to correct intercorrelations among subscale scores for multiple sources of measurement error. Within the G-theory SEMs considered here, random sampling or exchangeability of measurement facet conditions (i.e., items and occasions) was operationalized by setting relevant factor loadings, variances, and uniquenesses equal to produce the same results obtained from parallel ANOVA designs (see, e.g., [16,17,50,[84][85][86][87][88][89]).…”

“…Informative future extensions of the multivariate SEMs illustrated here would be to (a) analyze them using broader demographic groups beyond the present sample of college students, which was heavily dominated by female and Caucasian participants; (b) apply the procedures to objectively and subjectively scored measures within achievement, aptitude, behavioral, psychomotor, physiological, and other affective domains; (c) incorporate additional designs with different combinations of crossed and nested facets and more than two measurement facets (see, e.g., [14,16,17,89]); (d) use estimation procedures, when warranted, to adjust for scale coarseness effects common when using binary or ordinal level data [13,15,16,49,50,84,88,89,[94][95][96]; and (e) derive global and cut-score-specific dependability coefficients when using data for criterion-referencing purposes [13,14,16,17,50,70,[87][88][89][95][96][97][98][99][100][101][102]. We encourage researchers and practitioners to take advantage of these techniques to develop better assessment procedures and more thoroughly evaluate the psychometric quality of results obtained from them.…”