Maximum information per time unit designs for continuous online item calibration

Abstract

Previous designs for online calibration have only considered examinees' responses to items. However, the use of response time, a useful metric that can easily be collected by a computer, has not yet been embedded in calibration designs. In this article we utilize response time to optimize the assignment of new items online, and accordingly propose two new adaptive designs. These are the D-optimal per expectation time unit design (D-ET) and the D-optimal per time unit design (D-T). The former method uses the conditional maximum likelihood estimation (CMLE) method to estimate the expected response times, while the latter employs the nonparametric k-nearest-neighbour method to predict the response times. Simulations were conducted to compare the two new designs with the D-optimal online calibration design (D design) in the context of continuous online calibration. In addition, a preliminary study was carried out to evaluate the performance of CMLE prior to its application in D-ET. The results showed that, compared to the D design, the D-ET and D-T designs saved response time and accrued larger calibration information per time unit, without sacrificing item calibration precision.

Using Response Time in Multidimensional Computerized Adaptive Testing

“…Specifically, the item discrimination parameters were drawn from the uniform distribution $$U( {0,\;3} )$$ (Wang, 2015), the time discrimination parameters α were simulated from $$U( {2,\;4} )$$ (Fan et al., 2012), and the guessing parameters c were generated from $$U( {0,\;.2} )$$ (Wang, 2019). In a similar vein to the unidimensional situation (Choe et al., 2018; Fan et al., 2012; He et al., 2021; Wang, 2019), the item difficulty parameters $${b_j}$$ and time intensity parameters $${\beta _j}$$ were simulated from the multivariate normal distribution $$( {{b_j},\;{\beta _j}} )\sim MVN( {{\bm \mu} ,\;{\bm \Sigma} } )$$, where $${\bm \mu} = ( {0,\;0} )$$ is the mean vector and $${\bm \Sigma} = \left( { \def\eqcellsep{&}\begin{array}{@{}*{2}{c}@{}} 1&{.25}\\ {.25}&{.25} \end{array} } \right)$$ is the covariance matrix. As pointed out by Fan et al.…”

“…Hence, the parameters needed to be estimated are σ 12 , σ 13 , σ 23 , and $$\sigma _{33}^2$$ in the second level and the parameters in the first level. In adaptive testing, prior during a careful calibration study, the parameters of σ 12 , σ 13 , σ 23 , and $$\sigma _{33}^2$$ are usually assumed to be well estimated with sufficient precision to treat them as known, and the same assumption is made for the IRT item parameters $${\vec {\bm a}_j}$$, $${b_j}$$, and $${c_j}$$, and the lognormal response time parameters $${\alpha _j}$$ and $${\beta _j}$$ (e.g., Chen et al., 2017; Fan et al., 2012; He et al., 2021; van der Linden, 2008; van der Linden, 2011b). Hence, to adaptively administer items in CAT, the ability and speed parameters should be estimated.…”

“…Specifically, the item discrimination parameters were drawn from the uniform distribution U (0, 3) (Wang, 2015), the time discrimination parameters α were simulated from U (2, 4) (Fan et al, 2012), and the guessing parameters c were generated from U (0, .2) (Wang, 2019). In a similar vein to the unidimensional situation (Choe et al, 2018;Fan et al, 2012;He et al, 2021;Wang, 2019), the item difficulty parameters b j and time intensity parameters β j were simulated from the multivariate normal distribution (b j , β j ) ∼ MV N (μ, ), where μ = (0, 0) is the mean vector and = 1 .25 .25 .25 is the covariance matrix.…”

“…To broaden the applicability of methods focused on improving CAT delivery in terms of DTE, Cheng et al (2017) further modified the MIT criterion to a simple version (MIT-S) by using the expected logarithm of response time. Recently, the MIT framework was applied to adaptively measure individual change (Finkelman & Wang, 2019) and to calibrate items online (He et al, 2021), and was extended to the context of computerized classification testing (Sie et al, 2015) and cognitive diagnostic computerized adaptive testing (Huang, 2020). The results showed that items selected in the MIT framework saved testing time, ensured satisfactory measurement precision, and produced a higher true positive rate for measuring individual change.…”

“…With the aid of a computer, it is convenient to collect the response times spent by examinees on each item. Response time has been proven to be useful auxiliary information in the field of education measurement, such as detecting abnormal behaviors (Lu et al., 2020; Qian et al., 2016; Sinharay & Johnson, 2020; van der Linden, 2009a; van der Linden & Guo, 2008; Wang & Xu, 2015; Wang et al., 2018), detecting or controlling test speededness (van der Linden, 2009b; van der Linden, 2011b; van der Linden et al., 1999; van der Linden et al., 2007; Wang, 2019), and improving the estimation accuracy of the model parameters (He et al., 2021; Kang et al., 2020).…”