Practical issues in Large-Scale Computerized Adaptive Testing

Mills, Craig N.; Stocking, Martha L.

doi:10.1207/s15324818ame0904_1

Cited by 52 publications

(33 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Ideally, all items in a pool should have similar exposure rates to meet the requirements of test security and ef®ciency of item usage as well Mills & Stocking, 1996). Once items are developed, they have to satisfy particular speci®cations in order to form the required substantive and statistical distributions.…”

Section: Introductionmentioning

confidence: 99%

a‐Stratified CAT design with content blocking

Yi¹,

Chang

2003

Brit J Math & Statis

View full text Add to dashboard Cite

Content balancing is often required in the development and implementation of computerized adaptive tests (CATs). In the current study, we propose a modified a-stratified method, the a-stratified method with content blocking. As a further refinement of a-stratified CAT designs, the new method incorporates content specifications into item pool stratification. Simulation studies were conducted to compare the new method with three previous item selection methods: the a-stratified method; the a-stratified with b-blocking method; and the maximum Fisher information method with Sympson-Hetter exposure control. The results indicated that the refined a-stratified design performed well in reducing item overexposure rates, balancing item usage within the pool, and maintaining measurement precision, in a situation where all four procedures were forced to balance content.

show abstract

Section: Introductionmentioning

confidence: 99%

a‐Stratified CAT design with content blocking

Yi¹,

Chang

2003

Brit J Math & Statis

View full text Add to dashboard Cite

show abstract

“…To our knowledge, this question has only been addressed with respect to the initial item on a computer adaptive test. Although there is no evidence that the difficulty of the initial item affects performance on the test as a whole (Lunz, Bergstrom, & Gershon, 1994), educators have worried that reactions to the initial question may affect test-takers' perceptions of the test (Mills & Stocking, 1996). The one article that directly tested this hypothesis did not find evidence for any influence of the initial question on subsequent performance evaluations (Tonidandel, Quiñones, & Adams, 2002).…”

mentioning

confidence: 99%

Retrospective bias in test performance: Providing easy items at the beginning of a test makes students believe they did better on it

Weinstein

Roediger

2010

Memory & Cognition

View full text Add to dashboard Cite

We examined the effect of three variables (test list structure, report option, and framing) on retrospective bias in global evaluations of test performance (postdictions). Participants answered general knowledge questions and estimated correctness of their performance after each block. The ordering of the questions within a block affected bias: Participants believed they had answered more questions correctly when questions were sorted from the easiest to the hardest than when the same questions were randomized or sorted from the hardest to the easiest. This bias was obtained on global postdictions but was not apparent on item-by-item ratings, pointing to a memory-based phenomenon. In addition, forcing participants to produce a response to every question increased performance without affecting evaluations. Finally, framing the evaluation question in terms of the number of questions answered incorrectly (rather than the number correctly answered) did not affect how positively participants evaluated their performance, but did render postdictions less accurate. Our results provide evidence that students' evaluations of performance after a test are prone to retrospective memory biases.

show abstract

“…Average item overlap is defined by Way as the proportion (or percentage) of items shared by pairs of exams, averaged across all possible pairwise comparisons. It is important to note that Mills and Stocking (1996) use the term item overlap in referring to "the extent to which one item may cue the correct response to another item or the extent to which two items depend on the same specific knowledge" (p. 294). Because it is difficult to operationalize, item overlap, as defined by Mills and Stocking, is not considered here.…”

mentioning

confidence: 99%

The Relationship Between Item Exposure and Test Overlap in Computerized Adaptive Testing

Chen

Ankenmann

Spray³

2003

J Educational Measurement

View full text Add to dashboard Cite

The purpose of this article is to present an analytical derivation for the mathematical form of an average between-test overlap index as a fitnction of the item exposure index, for fixed-length computerized aahptive tests (CATs). This algebraic relationship is used to investigate the simultaneous control of item exposure at both the item and test levels. The results indicate that, in jixed-length CATs, control of the average between-test overlap is achieved via the mean and variance of the item exposure rates of the items that constitute the CAT item pool. The mean of the item exposure rates is easily manipulated Control over the variance of the item exposure rates can be achieved via the maximum item exposure rate (r-).Therefore, item exposure control methods which implement a specification of r, , (e.g., Sympson & Hetter;1985) provide the most direct control at both the item and test levels.

show abstract

Practical issues in Large-Scale Computerized Adaptive Testing

Cited by 52 publications

References 10 publications

a‐Stratified CAT design with content blocking

a‐Stratified CAT design with content blocking

Retrospective bias in test performance: Providing easy items at the beginning of a test makes students believe they did better on it

The Relationship Between Item Exposure and Test Overlap in Computerized Adaptive Testing

Contact Info

Product

Resources

About