IRT Approaches to Modeling Scores on Mixed‐Format Tests

Abstract

This article considers psychometric properties of composite raw scores and transformed scale scores on mixed‐format tests that consist of a mixture of multiple‐choice and free‐response items. Test scores on several mixed‐format tests are evaluated with respect to conditional and overall standard errors of measurement, score reliability, and classification consistency and accuracy under three item response theory (IRT) frameworks: unidimensional IRT (UIRT), simple structure multidimensional IRT (SS‐MIRT), and bifactor multidimensional IRT (BF‐MIRT) models. Illustrative examples are presented using data from three mixed‐format exams with various levels of format effects. In general, the two MIRT models produced similar results, while the UIRT model resulted in consistently lower estimates of reliability and classification consistency/accuracy indices compared to the MIRT models.

Reading is a multidimensional construct at child-L2-English-literacy onset, but comprises fewer dimensions over time: Evidence from multidimensional IRT analysis

“…The findings showed that the bi-factor MIRT model offered a better solution for Grade 1 in terms of model fit as indicated by the lower −2LL value, and model parsimony as indicated by the lower AIC, AICc, BIC, and SABIC values. This result is consistent with previous bi-factor MIRT applications in language assessment (Cai & Kunnan, 2018; Lee et al, 2019; Min & He, 2014), which also reported relatively better model–data fit of the bi-factor MIRT model over unidimensional IRT models and/or multidimensional generalizations of unidimensional IRT models. However, for Grades 2–12, the fit indices and model parsimony indicators tended to lead to different conclusions.…”

Two IRT Characteristic Curve Linking Methods Weighted by Information

“…First, as the proposed methods were developed and applied for dichotomous items only, future research would involve applying them to cases with polytomous or mixed‐format items. These methods could also be extended to more complex IRT models such as multidimensional models (e.g., Kim et al., 2020; Lee et al., 2020). Second, a future simulation study would involve study factors that were not considered in the present study, such as extremely small samples (e.g., Kolen, 2020; Peabody, 2020), vertical scaling, and so on.…”

Several Variations of Simple‐Structure MIRT Equating

“…Unlike UNT, true‐score equating under the SS‐MIRT framework also requires a multivariate true‐score distribution (see Equation ). Three potential methods appeared in the literature (Lee et al., 2020) to approximate the ability distribution of θ : (a) a quadrature distribution (D‐method), (b) Monte‐Carlo simulation (M‐method), and (c) individual latent‐trait estimates $$\widehat{\bm{\theta}}$$ (P‐method). In Lee et al.…”

“…In Lee et al. (2020), these three methods were discussed as a means to analyze psychometrics properties using MIRT models. However, they have not been used or introduced in the equating literature yet.…”