An Empirical Study of the Broad Range Tailored Test of Verbal Ability

Abstract

This document is the final report of a study designed to investigate the performance of the Broad-Range Tailored Test of Verbal Ability. The Broad-Range Tailored Test (BRTT) is a computerized adaptive test developed by Frederic M. Lord. It employs a maximum likelihood selection strategy to choose items from an item pool stored on magnetic disk. The items selected are tailored to the individual testee and are presented on a computer terminal.Each testee responds to 25 items; at the conclusion of the test the computer calculates a verbal ability score for the individual. The test was designed to yield a verbal ability score from the fifth grade level to the graduate school level.Performance of the BRTT had been investigated by means of simulation studies.The current study is the first empirical test of its performance. Two forms of the BRTT were administered to 146 high school students. The students also answered a posttest questionnaire in which they indicated their reactions to this form of testing.Analyses revealed that the BRTT wa s more reliable than the PSAT for a number of scores derived from the data. The test-retest reliability of the BRTT was .8719 at the 25th item; reliability of the PSAT v~rbal score (scaled down to 25 items) was .65. AIlalyses of the reliability of the BRTT vs. the PSAT revealed that the tailored test was also more reliable than the conventional test at shorter lengths.Correlations between scores on the BRTT and PSAT were reasonably high--typically about .86. This finding confirms theoretical expectation regarding the increased efficiency of adaptive, as compared to conventional tests.The study investigated nine of observed scores and score transformations. The most useful of these was found to be the expected proportion correct over the entire item p0ol. This score was highly reliable and was found to be parallel with respect to the mean values across forms A and B.e , the commonly-employed latent-trait parameter and :;"l, a monotone transformation of 8, did not exhibit this characteristic.The information functions of the BRTT were calculated and compared favorably with simulation results previously reported by Lord. Thus the accuracy of the BRTT was in accord with theoretical expectation. Student response to the computerized testing procedure was generally quite favorable.Students found the human-computer interface easy to use and less fatiguing than a long pencil-and-paper test.Operationally, the system performed well. Detailed analysis of 11 anomalous cases, suggested refinements to the system. Response time was adequate and consistent. Reliability of the hardware and software was excellent. These results suggest that computerized adaptive testing is iii ready to take the first steps out of the laboratory environment and find its place in the educational community.The recommendations emerging from this study are: (1) that the organization collaborate with an interested client to develop an adaptive test for use in an educational setting; (2) that the potential for micropr...

Applying computerized adaptive testing to the Desires for Speed Questionnaire in the Chinese population: A simulation study.

“…In the CAT, item selection is based on the previous response of a participant. However, we had nothing prior information about the individual at the beginning (Kreitzberg & Jones, 1980) and a provisional estimate of the latent trait is needed for the selection of this item. Smits et al (2011) and Yudien et al (2019) selected the first item that is most informative for the average latent trait score.…”

Computerized Adaptive Testing for Schizotypal Personality Disorder: Detecting Individuals at Risk

“…The item selection in CAT relies on participants’ response to previous items. Yet, the respondent knows nothing about prior information in the initial stage of a test ( Kreitzberg and Jones, 1980 ). Choosing an item randomly from the final item pool is an effective and uncomplicated method ( Magis and Barrada, 2017 ).…”

Development and Validation of an Item Bank for Depression Screening in the Chinese Population Using Computer Adaptive Testing: A Simulation Study

“…In the CAT simulation, item selection is dependent on the examinee's responses to a given item. However, the examinee knows nothing about prior information at the beginning (Kreitzberg and Jones, 1980 ). The first item was randomly selected from the final depression item bank (Magis and Barrada, 2017 ), as this method is simple and effective.…”