An Investigation of Item Calibration Methods in Multistage Testing

Cai, Liuhan; Albano, Anthony D.; Roussos, Louis A.

doi:10.1080/15366367.2021.1878778

Cited by 4 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, it is recommended to design the routing module in a wide range. Similar to the results of the current study, Cai et al (2021) state that misrouting decreases when the amount of module information of the routing module increases. In addition, Şahin (2020) concluded that the change in the difference between the b parameters does not make a significant difference on the measurement accuracy.…”

Section: Discussionsupporting

confidence: 89%

Bireyselleştirilmiş Çok Aşamalı Testlerde Test Tasarımının Yanlış Yönlendirmeye Etkisi

YİĞİTER,

DOĞAN

2023

Uluslararası Türk Eğitim Bilimleri Dergisi

View full text Add to dashboard Cite

Computerized Multistage Testing (MST) is an adaptive testing approach in which the test taker completes stages and modules on a pre-assembled panel according to his/her ability level. In MST, the test taker is routed to a module in the following stage based on his/her responses to the module in each stage. The test taker is expected to be routed to the module that fits his/her ability level best in the following stages. If the test taker is not routed to the module appropriate to his/her ability level, misrouting can be mentioned. Misrouting is thought to affect both measurement accuracy and the test taker's psychology. Although it is very difficult to completely eliminate misrouting, it is assumed that it can be reduced with the basic components of the MST design. The purpose of this study is to determine the level of misrouting according to different MST designs and to investigate the effects of changes in test design on the level of misrouting. The main components that are considered to affect misrouting are the MST test design [1-3, 1-2-3, 1-3-3], routing module design [Wide, Narrow], test length [12, 24, 36] and module length [L-S, M-M, S-L]. This study, which aims to reveal the current situation, is descriptive research and it was carried out by simulation method. The results of the study show that MST design and components can be effective in reducing misrouting. Three-stage MST designs offer lower misrouting and higher measurement accuracy than two-stage MST designs. Furthermore, increasing the test length and designing the routing module with a wide range of abilities reduce the misrouting rate. According to the measurement accuracy results, it can be stated that misrouting is not a significant problem in the MST in general, although the measurement accuracy of the misrouted test takers is low. It was concluded that the ability levels of the misrouted test takers were generally concentrated at the intersection points of the module information functions of the adjacent modules and generally in the middle of the ability scale.

show abstract

Section: Discussionsupporting

confidence: 89%

Bireyselleştirilmiş Çok Aşamalı Testlerde Test Tasarımının Yanlış Yönlendirmeye Etkisi

YİĞİTER,

DOĞAN

2023

Uluslararası Türk Eğitim Bilimleri Dergisi

View full text Add to dashboard Cite

show abstract

“…Field test data can be used to place pretest item parameter estimates on the scale of existing items in the item pool through various calibration and linking methods. Previous studies have focused on general calibration issues in MST that are not directly applicable to calibrating pretest items (e.g., Cai et al., 2021; Steinfeld & Robitzsch, 2021). Studies addressing pretest item calibration in the context of CAT or fixed‐test forms using various parameter calibration methods (e.g., Kang et al., 2017; Ban et al., 2001; Ban et al., 2002) are informative, but not directly applicable to MST.…”

Section: Pretest Item Calibration In Multistage Adaptive Testingmentioning

confidence: 99%

Pretest Item Calibration in Computerized Multistage Adaptive Testing

Erşen

Lee

2023

J Educational Measurement

View full text Add to dashboard Cite

The purpose of this study was to compare calibration and linking methods for placing pretest item parameter estimates on the item pool scale in a 1‐3 computerized multistage adaptive testing design in terms of item parameter recovery. Two models were used: embedded‐section, in which pretest items were administered within a separate module, and embedded‐items, in which pretest items were distributed across operational modules. The calibration methods were separate calibration with linking (SC) and fixed calibration (FC) with three parallel approaches under each (FC‐1 and SC‐1; FC‐2 and SC‐2; and FC‐3 and SC‐3). The FC‐1 and SC‐1 used only operational items in the routing module to link pretest items. The FC‐2 and SC‐2 also used only operational items in the routing module for linking, but in addition, the operational items in second stage modules were freely estimated. The FC‐3 and SC‐3 used operational items in all modules to link pretest items. The third calibration approach (i.e., FC‐3 and SC‐3) yielded the best results. For all three approaches, SC outperformed FC in all study conditions which were module length, sample size and examinee distributions.

show abstract

“…MSAT is an exam administration method that has recently become popular in assessment (Cai et al, 2021;Li et al, 2021;MacGregor et al, 2022;Shin et al, 2021). Test administration using the MSAT model can increase measurement efficiency (Berger et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

The Use of Stocking-Lord and Haebara Methods in Horizontal Equating: A Case of Indonesian Madrasah Competence Assessment

Kusaeri,

Ridho,

Wahyudi

2024

JP3I

View full text Add to dashboard Cite

Indonesian Madrasah Competence Assessment (AKMI) is a national assessment implemented each year held by the Ministry of Religious Affairs. One of the uniqueness of the AKMI is the use of different tests every year. AKMI focuses on capturing the development of learning in Madrasa by comparing the test scores of the current year with the previous year. An equating process is crucial for valid results when comparing scores. Therefore, this research aims to (a) equate the scientific literacy assessment tools at AKMI in 2022 with 2023 and (b) evaluate the business process of developing AKMI scientific literacy instruments (along with the MSAT design), which has implications for the equating process. This study adopted a Non-Equivalent Anchor Test (NEAT) design because the two test sets were parallel years, and the participants were from a diverse population. The data is from the AKMI Science Literacy of the Ministry of Religious Affairs, with 303,987 participants in 2022 and 342,987 in 2023 from the Islamic elementary school level. A total of 674 scientific literacy instrument items in 2022 and 1,392 items in 2023, with 90 items used as anchor items. There are 3 stages of analysis: pre-equalization, equalization calibration, and post-equalization analysis. The results show that there are differences in item parameter estimation results between 2022 and 2023, where 2022 has a higher level of item difficulty. Furthermore, the Stocking-Lord and Haebara methods had proven to be effective and had produced estimates with minimal differences in the equating process. In addition, the anchor items used as the basis for the equating do not represent the items as a whole in the item pool. These findings indicate the need for firm, careful standardization based on psychometric principles of the process at AKMI, from developing items to assembling items, testing, determining anchor items, and assembling items in the MSAT application.

show abstract

An Investigation of Item Calibration Methods in Multistage Testing

Cited by 4 publications

References 15 publications

Bireyselleştirilmiş Çok Aşamalı Testlerde Test Tasarımının Yanlış Yönlendirmeye Etkisi

Bireyselleştirilmiş Çok Aşamalı Testlerde Test Tasarımının Yanlış Yönlendirmeye Etkisi

Pretest Item Calibration in Computerized Multistage Adaptive Testing

The Use of Stocking-Lord and Haebara Methods in Horizontal Equating: A Case of Indonesian Madrasah Competence Assessment

Contact Info

Product

Resources

About