Item Selection Criteria With Practical Constraints for Computerized Classification Testing

Lin, Chuan-Ju

doi:10.1177/0013164410387336

Cited by 10 publications

(19 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results given in Table 1 and Figures 1-4 clearly show that SPRT improves test efficiency by shortening the test length by 19.6% without losing much capacity to maintain high classification accuracy, which is 93.7% here. A similar conclusion that SPRT tends to gives a better performance in CCT can be found in other work (Babcock & Weiss, 2009;Lin, 2011;Thompson, 2009) as well. Thus, SPRT is shown to be an efficient classification method which is used in simulations 2 and 3 for a further evaluation of the LA-CB methods.…”

Section: Results Of Simulationsupporting

confidence: 87%

Look‐ahead content balancing method in variable‐length computerized classification testing

Xiao

Zhang

Chang

2019

Brit J Math & Statis

View full text Add to dashboard Cite

Content balancing is one of the most important issues in computerized classification testing. To adapt to variable-length forms, special treatments are needed to successfully control content constraints without knowledge of test length during the test. To this end, we propose the notions of 'look-ahead' and 'step size' to adaptively control content constraints in each item selection step. The step size gives a prediction of the number of items to be selected at the current stage, that is, how far we will look ahead. Two look-ahead content balancing (LA-CB) methods, one with a constant step size and another with an adaptive step size, are proposed as feasible solutions to balancing content areas in variable-length computerized classification testing. The proposed LA-CB methods are compared with conventional item selection methods in variable-length tests and are examined with different classification methods. Simulation results show that, integrated with heuristic item selection methods, the proposed LA-CB methods result in fewer constraint violations and can maintain higher classification accuracy. In addition, the LA-CB method with an adaptive step size outperforms that with a constant step size in content management. Furthermore, the LA-CB methods generate higher test efficiency while using the sequential probability ratio test classification method.

show abstract

Section: Results Of Simulationsupporting

confidence: 87%

Look‐ahead content balancing method in variable‐length computerized classification testing

Xiao

Zhang

Chang

2019

Brit J Math & Statis

View full text Add to dashboard Cite

show abstract

“…In contrast to conclusions from our second simulation, many authors still take for granted the oft-repeated maxim that items should be selected by only considering the classification bound/bounds (e.g., Eggen, 2011;Finkelman, 2008a;Huebner, 2012;Lin, 2011;Lin & Spray, 2000;Wiberg, 2003). Up until now, this maxim was based on logically sound reasoning.…”

Section: Resultsmentioning

confidence: 75%

“…Note that Equation 23 is similar to Equation 21 with p j ðŷ i Þ replaced by p j ðy 0 þ dÞ. Although KL divergence uses a version of the SPRT-based loglikelihood ratio, researchers have found that selecting items by maximizing KL divergence does not result in shorter nor more accurate classification tests than selecting items by maximizing FI at y 0 (e.g., Eggen, 1999;Lin, 2011;Lin Nydick & Spray, 2000). A common complaint in using pointwise KL divergence to select items in CCT is the lack of symmetry between KL j ðy u jjy l Þ and KL j ðy l jjy u Þ.…”

Section: Simulationmentioning

confidence: 99%

The Sequential Probability Ratio Test and Binary Item Response Models

Nydick

2014

Journal of Educational and Behavioral Statistics

View full text Add to dashboard Cite

The sequential probability ratio test (SPRT) is a common method for terminating item response theory (IRT)-based adaptive classification tests. To decide whether a classification test should stop, the SPRT compares a simple log-likelihood ratio, based on the classification bound separating two categories, to prespecified critical values. As has been previously noted (Spray & Reckase, 1994), the SPRT test statistic is not necessarily monotonic with respect to the classification bound when item response functions have nonzero lower asymptotes. Because of nonmonotonicity, several researchers (including Spray & Reckase, 1994) have recommended selecting items at the classification bound rather than the current ability estimate when terminating SPRT-based classification tests. Unfortunately, this well-worn advice is a bit too simplistic. Items yielding optimal evidence for classification depend on the IRT model, item parameters, and location of an examinee with respect to the classification bound. The current study illustrates, in depth, the relationship between the SPRT test statistic and classification evidence in binary IRT models. Unlike earlier studies, we examine the form of the SPRT-based log-likelihood ratio while altering the classification bound and item difficulty. These investigations motivate a novel item selection algorithm based on optimizing the expected SPRT criterion given the current ability estimate. The new expected log-likelihood ratio algorithm results in test lengths noticeably shorter than current, commonly used algorithms, and with no loss in classification accuracy.

show abstract

“…Chen, Ankenmann, and Chang (2000) used KL information in the early stage of CAT. Eggen (1999) and Lin (2011) have applied KL information to computerized classification testing. Henson and Douglas (2005) proposed KL-based divergence indices to assemble cognitive diagnostic tests.…”

Section: Methodsmentioning

confidence: 99%

Item Selection Criteria With Practical Constraints in Cognitive Diagnostic Computerized Adaptive Testing

Lin

Chang

2018

Educational and Psychological Measurement

Self Cite

View full text Add to dashboard Cite

For item selection in cognitive diagnostic computerized adaptive testing (CD-CAT), ideally, a single item selection index should be created to simultaneously regulate precision, exposure status, and attribute balancing. For this purpose, in this study, we first proposed an attribute-balanced item selection criterion, namely, the standardized weighted deviation global discrimination index (SWDGDI), and subsequently formulated the constrained progressive index (CP_SWDGDI) by casting the SWDGDI in a progressive algorithm. A simulation study revealed that the SWDGDI method was effective in balancing attribute coverage and the CP_SWDGDI method was able to simultaneously balance attribute coverage and item pool usage while maintaining acceptable estimation precision. This research also demonstrates the advantage of a relatively low number of attributes in CD-CAT applications.

show abstract

Item Selection Criteria With Practical Constraints for Computerized Classification Testing

Cited by 10 publications

References 23 publications

Look‐ahead content balancing method in variable‐length computerized classification testing

Look‐ahead content balancing method in variable‐length computerized classification testing

The Sequential Probability Ratio Test and Binary Item Response Models

Item Selection Criteria With Practical Constraints in Cognitive Diagnostic Computerized Adaptive Testing

Contact Info

Product

Resources

About