Improving generalizability coefficient estimate accuracy: A way to incorporate auxiliary information

Abstract

Initially proposed by Marcoulides and further expanded by Raykov and Marcoulides, a structural equation modeling approach can be used in generalizability theory estimation. This article examines the utility of incorporating auxiliary variables into the structural equation modeling approach when missing data is present. In particular, the authors assert that by adapting a saturated correlates model strategy to structural equation modeling generalizability theory models, one can reduce any biased effects caused by missingness. Traditional approaches such as an analysis of variance do not possess such a feature. This article provides detailed instructions for adding auxiliary variables into a structural equation modeling generalizability theory model, demonstrates the corresponding benefits of bias reduction in generalizability coefficient estimate via simulations, and discusses issues relevant to the proposed approach.

How to Estimate Absolute-Error Components in Structural Equation Models of Generalizability Theory

“…A path diagram for this p × r × t design would resemble Figure 2, but (a) two tasks would replace three items and (b) six raters would replace three occasions. As the example syntax on OSF shows, a properly specified SEM for this GT design does not converge on a ML solution when fit to the example data (also provided on OSF) using full-information ML (FIML) to accommodate incomplete data [19]. Rather than a problem with estimating model parameters, the lavaan package warned that the "maximum number of iterations reached when computing the sample moments using EM" (i.e., not all sample covariances could be estimated), although the error did not indicate that 0% coverage absolutely prevented estimating the model parameters.…”

“…Given the frequent use of factor analysis in scale development, specifying GT models as CFAs brings together two very useful frameworks for evaluating measurement instruments and procedures. When measurements include unplanned missing data-for which the MCAR assumption is unlikely to be met-incorporating auxiliary variables into a saturated-correlates model can make the less restrictive missing-at-random (MAR) assumption easier to justify [19]. Although the examples in this article have respected GT's traditional assumption of randomly parallel measures, SEM enables assumptions to be relaxed or tested against the data, such as homoskedasticity across measurement conditions [16,17] and congeneric measurement [25] (i.e., unequal factor loadings).…”

“…The method also becomes infeasible in the very common case that there are more rows (subjects) of data than columns (measurement conditions). To correct a possible misconception that the drawbacks of the Q-method would limit a researcher to computing a relative G-coef [18,19], I show how absolute-error variance components can be indirectly captured as functions of parameters in the same SEM that estimates relative-error variance components.…”

Particle Swarm Optimization for Diagnosis Task: An Exploratory Study

“…Beyond the scope of the present paper, a large-scale simulation may be needed to guide parameter tunning more sysmetically and efficiently. In addition, the missing data problem is not concerned here; in the future study, incorporating auxiliary information into the algorithm may help reduce estimation bias [23].…”

Using a linear mixed-effect model framework to estimate multivariate generalizability theory parameters in R

“…As a frequentist product, mGENOVA is an ANSI C program powered by mean square error calculations, which are typically seen in traditional repeated-measures ANOVA estimations. These ANOVA approaches can also be realized in a structural equation modeling framework (Jiang, Walker, Shi, & Cao, 2018;Marcoulides, 1996;Raykov & Marcoulides, 2006). On the other hand, BUGS/JAGS refers to three variants sharing highly similar algorithmic functionalities: WinBUGS (Lunn, Thomas, Best, & Spiegelhalter 2000;Spiegelhalter, Thomas, Best, & Lunn 2003), OpenBUGS (Thomas, O'Hara, Ligges, & Sturtz 2006), and JAGS (Plummer 2003(Plummer , 2010.…”