Efficient Standard Errors in Item Response Theory Models for Short Tests

Ippel, Lianne; Magis, David

doi:10.1177/0013164419882072

Cited by 3 publications

(2 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In other words, this is a decent approximation of SE for sufficiently long tests only. In fact, the inverse Fisher information tends to overestimate the variance of

\hat{\theta }_{\mathrm{ML}}

and

\hat{\theta }_{\mathrm{WL}}

for short tests especially at the extremes (e.g., Magis, 2014), so exact SEs and confidence intervals have been proposed in recent literature (e.g., Doebler, Doebler, & Holling, 2013; Ippel & Magis, 2020; Liu et al., 2018; Magis, 2016). Furthermore, the SE of boundary estimates are not well understood, because the asymptotic properties of ML and WL only hold under certain regularity conditions, one of which is that the estimated parameter must be an interior point of the parameter space.…”

Section: Practical Implications and Conclusionmentioning

confidence: 99%

Constructing a Robust Score Scale from IRT Scores with Informed Boundaries

Choe¹,

Han²

2022

J Educational Measurement

View full text Add to dashboard Cite

In operational testing, item response theory (IRT) models for dichotomous responses are popular for measuring a single latent construct θ, such as cognitive ability in a content domain. Estimates of θ, also called IRT scores or θ, can be computed using estimators based on the likelihood function, such as maximum likelihood (ML), weighted likelihood (WL), maximum a posteriori (MAP), and expected a posteriori (EAP). Although the parameter space of θ is theoretically unrestricted, the range of finite θ is constrained by the estimator and test form properties, which is important to consider but often overlooked when developing a score scale for reporting purposes. Irrespective of the estimator or test forms at hand, a common practice is to fix arbitrary points symmetric about zero (e.g., −4 and 4) as anchors for deriving a score transformation, possibly resulting in unintended gaps or truncations at the extremes. Therefore, a systematic framework is proposed for using IRT scores to construct a robust score scale with informed boundaries that are logical and consistent across test forms.

show abstract

“…In other words, this is a decent approximation of SE for sufficiently long tests only. In fact, the inverse Fisher information tends to overestimate the variance of

\hat{\theta }_{\mathrm{ML}}

and

\hat{\theta }_{\mathrm{WL}}

Section: Practical Implications and Conclusionmentioning

confidence: 99%

Constructing a Robust Score Scale from IRT Scores with Informed Boundaries

Choe¹,

Han²

2022

J Educational Measurement

View full text Add to dashboard Cite

show abstract

“…However, in practice whether this estimated information is a good indicator for the SE of the ability parameter in IRT models is yet well-established. Ippel and Magis (2020) and Magis (2014) pointed out that the Fisher information might fail to precisely estimate the SE of the ability parameter when the item number is small. Moreover, to the best of our knowledge, the asymptotic property of the SE estimated by the Fisher information is still unclear, let alone whether it is the most asymptotically efficient estimator for the SE.…”

Section: Introductionmentioning

confidence: 99%

Parametric bootstrapping of standard errors for the ability parameter in IRT models: Asymptotic and small sample analyses

Yang,

Hsu,

Cheng

2024

Preprint

View full text Add to dashboard Cite

Assessing a participant’s latent ability is one of the goals of item response theory (IRT) and IRT-based computerized adaptive testing (CAT). In the literature, the commonly adopted indicator of the precision of the ability estimator in IRT (including CAT) is the Fisher information. However, this index cannot reliably reflect the precision of the estimate when the number of items is small. Two alternatives to evaluating the standard error (SE) of the ability estimator have been proposed: the Monte Carlo parametric bootstrap \parencite{Liou1991} and the ``exact SE'' method \parencite{magis2014}. The first part of this study contains some theoretical work. We start with showing that Magis's method is an instance of the parametric bootstrap. In particular, we derive the exact bootstrap distribution based on the principle of Magis's algorithm. Next, we investigate the asymptotic properties of the parametric bootstrap SE under the independent but not necessarily identically distributed condition. We prove the consistency of the parametric bootstrap SE. The asymptotic relative efficiency of it over the Fisher information SE is also established. The second part contains an improvement of the parametric-bootstrap algorithm for SE to reduce computational complexity. Using minimal sufficient statistics, we develop an accelerated version of the algorithm for the two-parameter logistic model. The last part concerns the small sample analysis. We conduct a series of simulations comparing the performance of the SE estimator from our proposed algorithm with that from the information-based method. We also illustrate an application of our algorithm on test construction using existing data with small item numbers. Overall, the results reveal an advantage of our newly developed method over the the information-based method for assessing the precision of the ability estimator in IRT.

show abstract

Estimating and Using Block Information in the Thurstonian IRT Model

Frick

2023

Psychometrika

View full text Add to dashboard Cite

Multidimensional forced-choice (MFC) tests are increasing in popularity but their construction is complex. The Thurstonian item response model (Thurstonian IRT model) is most often used to score MFC tests that contain dominance items. Currently, in a frequentist framework, information about the latent traits in the Thurstonian IRT model is computed for binary outcomes of pairwise comparisons, but this approach neglects stochastic dependencies. In this manuscript, it is shown how to estimate Fisher information on the block level. A simulation study showed that the observed and expected standard errors based on the block information were similarly accurate. When local dependencies for block sizes $$>\,2$$ > 2 were neglected, the standard errors were underestimated, except with the maximum a posteriori estimator. It is shown how the multidimensional block information can be summarized for test construction. A simulation study and an empirical application showed small differences between the block information summaries depending on the outcome considered. Thus, block information can aid the construction of reliable MFC tests.

show abstract

Efficient Standard Errors in Item Response Theory Models for Short Tests

Cited by 3 publications

References 31 publications

Constructing a Robust Score Scale from IRT Scores with Informed Boundaries

Constructing a Robust Score Scale from IRT Scores with Informed Boundaries

Parametric bootstrapping of standard errors for the ability parameter in IRT models: Asymptotic and small sample analyses

Estimating and Using Block Information in the Thurstonian IRT Model

Contact Info

Product

Resources

About