Exploring the Impact of Missing Data on Residual-Based Dimensionality Analysis for Measurement Models

Abstract

Researchers frequently use Rasch models to analyze survey responses because these models provide accurate parameter estimates for items and examinees when there are missing data. However, researchers have not fully considered how missing data affect the accuracy of dimensionality assessment in Rasch analyses such as principal components analysis (PCA) of standardized residuals. Because adherence to unidimensionality is a prerequisite for the appropriate interpretation and use of Rasch model results, insight into the impact of missing data on the accuracy of this approach is critical. We used a simulation study to examine the accuracy of standardized residual PCA with various proportions of missing data and multidimensionality. We also explored an adaptation of modified parallel analysis in combination with standardized residual PCA as a source of additional information about dimensionality when missing data are present. Our results suggested that missing data impact the accuracy of PCA on standardized residuals, and that the adaptation of modified parallel analysis provides useful supplementary information about dimensionality when there are missing data.

Item-Level Analysis of a Newly Developed Interactive Nutrition Specific Physical Exam Competency Tool (INSPECT) Using the Rasch Measurement Model

“…First, the PCA of Rasch residuals is for diagnostic purposes as mentioned before rather than for definitive conclusions of unidimensionality [ 36 , 44 ]. Second, the cut-off points are arbitrary and depend on the sample size and the number of items [ 46 ]. Third, as the INSPECT consists of several items grouped as subsets or subcomponents, there is potential for items within these subtests to be more correlated causing the increase in eigenvalue in the first contrast.…”

Developing and Validating an Instrument to Assess Ninth-Grade Students’ Online Metacognitive Skills in Solving Chemistry Problems

“…Unidimensionality. Unidimensionality is an explicit requirement for Rasch measurement (Smith, 1996), assuming that the test measures only one underlying construct (Wind & Schumacker, 2021). This study performed a principal component analysis of residuals to check for unidimensionality (Bond & Fox, 2015).…”