Effects of statistical models and items difficulties on making trait-level inferences: A simulation study

Abstract

Researchers dealing with the task of estimating locations of individuals on continuous latent variables may rely on several statistical models described in the literature. However, weighting costs and benefi ts of using one specifi c model over alternative models depends on empirical information that is not always clearly available. Therefore, the aim of this simulation study was to compare the performance of seven popular statistical models in providing adequate latent trait estimates in conditions of items diffi culties targeted at the sample mean or at the tails of the latent trait distribution. Results suggested an overall tendency of models to provide more accurate estimates of true latent scores when using items targeted at the sample mean of the latent trait distribution. Rating Scale Model, Graded Response Model, and Weighted Least Squares Mean-and Variance-adjusted Confi rmatory Factor Analysis yielded the most reliable latent trait estimates, even when applied to inadequate items for the sample distribution of the latent variable. These fi ndings have important implications concerning some popular methodological practices in Psychology and related areas. Keywords: Psychometrics, Item Response Theory, Classical Test Theory, factor analysis, data simulation, latent variable models. ResumoPesquisadores interessados em estimar a localização de indivíduos em variáveis latentes contínuas podem se benefi ciar de diversos modelos estatísticos disponíveis na literatura. Entretanto, ponderar os custos e os benefícios de usar um modelo em detrimento de outros depende de informações empíricas que nem sempre estão diretamente disponíveis. Em virtude disso, o objetivo deste estudo foi comparar o desempenho de sete modelos estatísticos populares quanto a proporcionar adequadas estimativas de traço latente em condições de itens com difi culdades condizentes com a distribuição latente amostral versus apenas condizentes com as caudas dessa distribuição. Os resultados sugeriram uma tendência de todos os modelos de proporcionar estimativas mais precisas ao serem usados itens adequados para o nível de traço latente da amostra. Os modelos da Teoria de Resposta ao Item Rating Scale e Graded Response e a análise fatorial confi rmatória com estimação Weighted Least Squares Mean-and Variance-adjusted forneceram as estimativas mais fi dedignas de traço latente, mesmo quando os itens utilizados, de fato, correspondiam ao nível latente de poucos casos da amostra. Os resultados possuem importantes implicações no que diz respeito a algumas práticas metodológicas populares na Psicologia e em áreas próximas. Palavras-chave: Psicometria, Teoria de Resposta ao Item, Teoria Clássica dos Testes, análise fatorial, simulação de dados, modelos de variáveis latentes.

Effects of Various Simulation Conditions on Latent-Trait Estimates: A Simulation Study

“…These values were obtained by means of the z-score table. These values are adapted from Hauck Filho, et al (2014).…”

“…For this purpose, in the beta distribution, value a was 2.9 and value b was 5.7. These values are adapted from Hauck Filho, et al (2014).…”

“…The difference in the sample size was determined, considering previous simulation studies (Dawber, Rogers, & Carbonaro, 2009;Hauck Filho, et al, 2014). Even though one of the factors affecting the psychometric traits of measurement instruments is the response formats (Jafari, Bagheri, Ayatollahi, & Soltani, 2012), the same number of response formats was used in almost all simulation studies.…”

“…Not many studies are encountered in the related literature which comparisons are made between the different parameter estimation methods on these techniques, namely CTT, IRT, and FA (Dumenci & Achenbach, 2008;Hauck Filho, Machado, & Damásio, 2014). In one study, conducted by Dumenci and Achenbach (2008), six statistical models that could estimate different latent traits were compared: CTT, PCA, CFA using maximum likelihood estimation, CFA using weighted least squares, graded response model (GRM) and partial credit model (PCM).…”

“…As real data were used in the study, the lack of control variables made it difficult for the models to be compared. In another study, conducted by Hauck Filho et al (2014), seven different statistical models that could estimate latent traits were compared: CTT, PCA, EFA using Maximum Likelihood, EFA with Minimum Rank, RSM, GRM and CFA with weighted least squares. This comparative study was performed with a total of 15 different simulative datasets comprised of three different item difficulty distributions and five different sample sizes.…”