An Investigation of the Use of Simplified Irt Models for Scaling and Equating the Toefl Test

Abstract

A continuing program of research related to the TOEFL test is carried out under the direction of the TOEFL Research Committee. Its six members include representatives of the Policy Council. the TOEFL Committee of Examiners, and distinguished English as a second language specialists from the academic community. Currently the Committee meets twice yearly to review and approve proposals for test-related research and to set guidelines for the entire scope of the TOEFL research program. Members of the Research Committee serve three-year terms at the invitation of the Policy Council; the chair of the committee serves on the Policy Council.Because the studies are specific to the test and the testing program, most of the actual research is conducted by ETS staff rather than by outside researchers. However, many projects require the cooperation of other institutions, particularly those with programs in the teaching of English as a foreign or second language. Representatives of such programs who are interested in participating in or conducting TOEFL-related research are invited to contact the TOEFL program office. Local research may sometimes require access to TOEFL data. In such cases, the program may provide the data following approval by the Research Committee. All TOEFL research projects must undergo appropriate ETS review to ascertain that the confidentiality of data will be protected. Current (1990-91) AbstractThe purpose of this study was to explore the use of two alternative item !esponse theory estimation models in the scaling and equating of TOEFL --a modified one-parameter model (MIPL) and a modified two-parameter model (M2PL) --and to compare item scaling and test equating results based on these two alternative models with results based on the threeparameter model (3PL) that is currently being used to scale and equate the TOEFL. The study employed a design in which a typical TOEFL equating was simulated using artificial data. The simulated equatings were compared in terms of correlations between estimated and generating parameters, model-data fit, and concordance of simulated score conversions with conversions based on the generating parameters.The results of the study dearly indicated that the 3PL model performed better than the MIPL and M2PL models on the basis of each of the evaluation criteria. There was also evidence that the M2PL model performed better than the MIPL model, particularly in terms of model-data fit and in the weighted root mean square difference statistics used to evaluate the simulated score conversions. The results of the study also indicated that discrepancies between score conversions based on the MIPL and M2PL model and those based on the 3PL model tended to occur at the lower and upper ends of the score scales. Finally, the results of the study for the 3PL model indicated that while correlations between item parameter estimates and generating parameters tended to be affected by sample size, neither the quality of model-data fit nor the quality of simulated equatings appeared to be ...

Cross‐validation of a Proportional Item Response Curve Model

“…But a previous TOEFL research study using the modified Rasch model was conducted by Hicks (1984) and another by Way and Reese (1991). Both of these studies compared the performance of this modified Rasch model with 3PL; and both found in favor of 3PL, in contrast with the present study.…”

“…The study by Way and Reese (1991) was a simulation study in which 3PL was used to generate simulated samples, and then the fit of other models to those samples was evaluated. In such a situation the 3PL model should be the favorite because it is approximating itself.…”

The Effect of Small Calibration Sample Sizes on Toefl Irt‐based Equating

“…It was assumed that enough similarities exist between the three sections of the TOEFL that results based on one of the sections would be relevant for the other two sections. Support for this assumption can be seen in the simulation results reported by Way and Reese (1991). The structure of the data for Simulation I was identical to the structure of the operational TOEFL data that were analyzed in the real data portion of the study.…”

“…However, in estimating the true score equating relationship, which is a common application at ETS, the two programs were about equally accurate when the number of items in a test was greater than or equal to 20. In a simulation study based on the TOEFL operational equating design, Way and Reese (1991) found that although correlations between LOGIST 3PL item parameter estimates and generating "true" parameters tended to be affected by sample size, neither the quality of model-data fit nor the quality of simulated equatings appeared to be sensitive to sample size. In their study, sample sizes for equating set items ranged from 600 to 1,500, while operational items were based on sample sizes ranging from 2,400 to 6,000.…”

“…The first simulation (Simulation I) applied the procedures used by Way and Reese (1991). That is, 3PL model item parameters estimated by LOGIST for the TOEFL April 1991 administration were used as the generating "true" item parameters for the simulated data.…”