Examination of Polytomous Items’ Psychometric Properties According to Nonparametric Item Response Theory Models in Different Test Conditions

International Journal of Assessment Tools in Education

2021

There is a great deal of research about item response theory (IRT) conducted by simulations. Item and ability parameters are estimated with varying numbers of replications under different test conditions. However, it is not clear what the appropriate number of replications should be. The aim of the current study is to develop guidelines for the adequate number of replications in conducting Monte Carlo simulation studies involving unidimensional IRT models. For this aim, 192 simulation conditions which included four sample sizes, two test lengths, eight replication numbers, and unidimensional IRT models were generated. Accuracy and precision of item and ability parameter estimations and model fit values were evaluated by considering the number of replications. In this context, for the item and ability parameters; mean error, root mean square error, standard error of estimates, and for model fit; 2 , 2 , and Type I error rates were considered. The number of replications did not seem to influence the model fit, it was decisive in Type I error inflation and error prediction accuracy for all IRT models. It was concluded that to get more accurate results, the number of replications should be at least 625 in terms of accuracy of the Type I error rate estimation for all IRT models. Also, 156 replications and above can be recommended. Item parameter biases were examined, and the largest bias values were obtained from the 3PL model. It can be concluded that the increase in the number of parameters estimated by the model resulted in more biased estimates.

Section: Study Design Factorsmentioning

confidence: 99%

A Guide for More Accurate and Precise Estimations in Simulative Unidimensional IRT Models

Pekmezci

International Journal of Assessment Tools in Education

2021

“…It is observed in the literature review that several studies in which the psychometric properties of the scales are examined according to the NIRT models are carried out in Turkey and abroad (Galindo Garre et al, 2014;Sachs, Law, and Chan, 2003;Koğar, 2015;Şengül, Avşar, and Tavşancıl, 2017;Young, Blodgett and Reardon, 2003). Especially in the studies on the polytomous items, researches are carried out on both simulated and real data sets according to various factors such as sample size, the number of items and distribution of sample.…”

Section: Extended Abstractmentioning

confidence: 99%

“…NIRT models are models that provide convenience in applications for short tests and small samples. Upon analysing the literature, it is seen that NIRT models can be classified as Mokken model and nonparametric regression models (Şengül Avşar and Tavşancıl, 2017). Mokken model is divided into sub-models as Monotone Homogeneity Model (MHM) and Double Monotonicity Model (DMM) (Sijtsma and Molenaar, 2002).…”

Section: Extended Abstractmentioning

confidence: 99%

“…It is seen clearly that MHM requires similar assumptions as unidimensional PIRT models. However, when the PIRT and NIRT models are compared, it is known that the essential difference between the two models depends on the item characteristic curves (ICC) (Şengül Avşar and Tavşancıl, 2017). While ICCs are monotonically and logistically estimated in PIRT models, these curves cannot be estimated in NIRT models logistically, even though they are monotone (Sijtsma and Molenaar, 2002).…”

Section: Extended Abstractmentioning

confidence: 99%

See 1 more Smart Citation

Kategori Sayısının Psikometrik Özellikler Üzerine Etkisinin Mokken Homojenlik Modeli’ne Göre İncelenmesi

Eğitimde Ve Psikolojide Ölçme Ve Değerlendirme Dergisi

2018

Öz Araştırmanın amacı çok kategorili puanlanan maddelerden oluşan testlerde kategori sayısının psikometrik özellikler üzerindeki etkisinin parametrik olmayan madde tepki kuramı (POMTK) modeli ile belirlenmesidir. Belirlenen amaç doğrultusunda iki farklı büyüklükte (100 ve 500), çeşitli dağılım özelliklerine sahip (normal dağılan, sağa çarpık dağılan ve sola çarpık dağılan) örneklemler için iki farklı test uzunluğunda (10 madde ve 30 madde), üç farklı sayıda kategoriye (üç, beş ve yedi) sahip maddeler simülatif olarak üretilmiştir. Kategori sayısının psikometrik özellikler üzerindeki etkisi POMTK modellerinden Mokken Homojenlik Modeli (MHM) ile araştırılmıştır. Yapılan araştırma temel araştırma olarak tasarlanmıştır. Verilerin üretilmesinde ve verilerin analizinde R Studio 3.4.0 yazılımı kullanılmıştır. R Studio yazılımında MHM'ye göre analizler Mokken paketi ile yapılmıştır. MHM'ye göre yapılan ölçekleme sonucunda kategori sayısının değişmesiyle birlikte maddelerin MHM'ye uyumunda belli bir örüntü gözlenmemiştir. Genel olarak hem kısa testlerde, hem de uzun testlerde kategori sayısının güvenirlik değerlerinin kestiriminde etkili olmadıkları gözlenmiştir. Araştırmada belirlenen test koşullarında testler MHM'ye düşük düzeyde uyumlu çıkmıştır.

“…It must be emphasized that in practice if researchers want to study aberrant response behavior with N-PFS, researcher should investigate MHM assumptions. MHM can fit with skewed data (Şengül Avşar & Tavşancıl, 2017). MHM is an appropriate model for small samples (Junker & Sijtsma, 2001;Molenaar, 2001).…”

mentioning

confidence: 99%

Comparison of Person-Fit Statistics for Polytomous Items in Different Test Conditions

Eğitimde Ve Psikolojide Ölçme Ve Değerlendirme Dergisi

2019

The validity of individual test scores is an important issue that needs to be studied in psychological and educational assessment. An important factor affecting the validity of individual test scores is aberrant item response behavior. Aberrant item scores may increase/decrease the individuals' scores and as a result individuals' ability can be estimated above/below their true ability. Person-fit statistics (PFS) are useful tools to detect aberrant behavior. There are a great number of parametric and nonparametric PFS in the literature. The general purpose of the study is to examine the effectiveness of the parametric and nonparametric PFS in data sets which consist of polytomous items. This study is fundamental research aimed at determining the effectiveness of PFS using simulated data sets. According to the results, as expected, as the Type I error rates (significance alpha level) increased, detection rates (power) increased. In general, it is seen that as the number of misfitting item score vector and number of items increased, detection rates increased. Generally, nonparametric PFS (N-PFS) (especially G P) detected more aberrant individuals than parametric PFS (P-PFS) lz p. However, in some tests' conditions lz p detected more aberrant individuals than N-PFS for longer tests. The results indicate that N-PFS outperformed P-PFS in most of the test conditions.