Data Sparseness and On‐Line Pretest Item Calibration‐Scaling Methods in CAT

Ban, Jae‐Chun; Hanson, Bradley A.; Yi, Qing; Harris, Deborah J.

doi:10.1111/j.1745-3984.2002.tb01174.x

Cited by 12 publications

(19 citation statements)

References 6 publications

(13 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, if CAT is primarily used to diagnose students' preparation for an upcoming exam in order to help them study, then there is less concern about security breaches. Finally, more advanced techniques, such as online calibration [52][53][54], could be embedded into the platform, thereby reducing the cost for expanding or replenishing the item pools. USING COMPUTER ADAPTIVE TESTING TO … PHYS.…”

Section: Discussionmentioning

confidence: 99%

Using computer adaptive testing to assess physics proficiency and improve exam performance in an introductory physics course

Morphew

Mestre

Kang

et al. 2018

Phys. Rev. Phys. Educ. Res.

View full text Add to dashboard Cite

Prior research has established that students often underprepare for midterm examinations yet remain overconfident in their proficiency. Research concerning the testing effect has demonstrated that utilizing testing as a study strategy leads to higher performance and more accurate confidence compared to more common study strategies such as rereading or reviewing homework problems. We report on three experiments that explore the viability of using computer adaptive testing (CAT) for assessing students' physics proficiency, for preparing students for midterm exams by diagnosing their weaknesses, and for predicting scores in midterm exams in an introductory calculus-based mechanics course for science and engineering majors. The first two experiments evaluated the reliability and validity of the CAT algorithm. In addition, we investigated the ability of the CAT test to predict performance on the midterm exam. The third experiment explored whether completing two CAT tests in the days before a midterm exam would facilitate performance on the midterm exam. Scores on the CAT tests and the midterm exams were significantly correlated and, on average, were not statistically different from each other. This provides evidence for moderate parallel-forms reliability and criterion-related validity of the CAT algorithm. In addition, when used as a diagnostic tool, CAT showed promise in helping students perform better on midterm exams. Finally, we found that the CAT tests predicted the average performance on the midterm exams reasonably well, however, the CAT tests were not as accurate as desired at predicting the performance of individual students. While CAT shows promise for practice testing, more research is needed to refine testing algorithms to increase reliability before implementing CAT for summative evaluations. In light of these findings, we believe that more research is needed comparing CAT to traditional paper-and-pencil practice tests in order to determine whether the effort needed to create a CAT system is worthwhile.

show abstract

Section: Discussionmentioning

confidence: 99%

Using computer adaptive testing to assess physics proficiency and improve exam performance in an introductory physics course

Morphew

Mestre

Kang

et al. 2018

Phys. Rev. Phys. Educ. Res.

View full text Add to dashboard Cite

show abstract

“…There has been different emphasis on the various aspects of item pretesting. Most studies have focused on the estimation methods (e.g., Ban, Hanson, Yi, & Harris, ; Ban et al, ). Neglected in the literature have been other aspects of item pretesting, such as pretest design, including the item selection rules for pretesting, especially in a testing mode like CAT.…”

Section: Item Calibration Within Computerized Adaptive Testing (Cat) mentioning

confidence: 99%

“…Though a standard error rule may be more economical, it is more convenient to use a sample size rule (e.g., Wang & Wiley, ; Zhu, ). Ban et al () compared three sample sizes, 300, 1,000, and 3,000, while Ban et al () used 500 as a sample size, which is a reasonable choice for the marginal maximum likelihood (MML) method. Zhu () chose three sample sizes for investigation: 300, 500, and 1,000.…”

Section: Item Calibration Within Computerized Adaptive Testing (Cat) mentioning

confidence: 99%

See 1 more Smart Citation

An Item‐Driven Adaptive Design for Calibrating Pretest Items

Ali

Chang

2014

ETS Research Report Series

View full text Add to dashboard Cite

Adaptive testing is advantageous in that it provides more efficient ability estimates with fewer items than linear testing does. Item‐driven adaptive pretesting may also offer similar advantages, and verification of such a hypothesis about item calibration was the main objective of this study. A suitability index (SI) was introduced to adaptively select pretest items, by which an easy‐to‐implement calibration methodology—adaptive design—can be used. A simulation study was conducted to evaluate the proposed adaptive design as compared to existing methodologies. Results indicate that the adaptive design has many desired features in item calibration, including less bias and more accurate parameter estimates, than the existing methods do. The SI is promising and flexible enough to apply additional constraints on the calibration sample and on the pretest items, for example, constraints on response time. It can also be used to try out individual item modules such as those used in multistage testing. Study limitations and future research are also covered in this report.

show abstract

“…Traditional statistical parameter estimation algorithms in item response theory (IRT) have been adapted into the novel setting of online calibration (e.g., Ban, Hanson, Wang, Yi, & Harris, 2001;Ban, Hanson, Yi, & Harris, 2002;Stocking, 1988;Wainer & Mislevy, 2000). The most popular estimation methods proposed for online calibration are the one EM cycle method (OEM; Wainer & Mislevy, 2000) and the multiple EM cycle method (MEM;Ban et al, 2001).…”

mentioning

confidence: 99%

Online Calibration of Polytomous Items Under the Generalized Partial Credit Model

Zheng

2016

Applied Psychological Measurement

View full text Add to dashboard Cite

Online calibration is a technology-enhanced architecture for item calibration in computerized adaptive tests (CATs). Many CATs are administered continuously over a long term and rely on large item banks. To ensure test validity, these item banks need to be frequently replenished with new items, and these new items need to be pretested before being used operationally. Online calibration dynamically embeds pretest items in operational tests and calibrates their parameters as response data are gradually obtained through the continuous test administration. This study extends existing formulas, procedures, and algorithms for dichotomous item response theory models to the generalized partial credit model, a popular model for items scored in more than two categories. A simulation study was conducted to investigate the developed algorithms and procedures under a variety of conditions, including two estimation algorithms, three pretest item selection methods, three seeding locations, two numbers of score categories, and three calibration sample sizes. Results demonstrated acceptable estimation accuracy of the two estimation algorithms in some of the simulated conditions. A variety of findings were also revealed for the interacted effects of included factors, and recommendations were made respectively.

show abstract

Data Sparseness and On‐Line Pretest Item Calibration‐Scaling Methods in CAT

Cited by 12 publications

References 6 publications

Using computer adaptive testing to assess physics proficiency and improve exam performance in an introductory physics course

Using computer adaptive testing to assess physics proficiency and improve exam performance in an introductory physics course

An Item‐Driven Adaptive Design for Calibrating Pretest Items

Online Calibration of Polytomous Items Under the Generalized Partial Credit Model

Contact Info

Product

Resources

About