A Randomized Experiment to Compare Conventional, Computerized, and Computerized Adaptive Administration of Ordinal Polytomous Attitude Items

Abstract

A total of 520 high school students were randomly assigned to a paper-and-pencil test (PPT), a computerized standard test (CST), or a computerized adaptive test (CAT) version of the Dutch School Attitude Questionnaire (SAQ), consisting of ordinal polytomous items. The CST administered items in the same order as the PPT. The CAT administered all items of three SAQ subscales in adaptive order using Samejima's graded response model, so that six different stopping rule settings could be applied afterwards. School marks were used as external criteria.Results showed significant but small multivariate administration mode effects on conventional raw scores and small to medium effects on maximum likelihood latent trait estimates. When the precision of CAT latent trait estimates decreased, correlations with grade point average in general decreased. However, the magnitude of the decrease was not very large as compared to the PPT, the CST, and the CAT without the stopping rule.

Computerized Adaptive Testing for Polytomous Motivation Items: Administration Mode Effects and a Comparison With Short Forms

“…Only very short CATs consisting of 1 or 2 items led to administration mode effects, but for these CATs, differences between the CT and the CAT group were limited to the variances. Hol et al (2005) compared a PPT, a CT, and a CAT in an experimental study, which also involved a motivation scale of 48 polytomous items. They detected small but inconsequential CAT administration mode effects, which however were greatest for the shortest CATs.…”

“…These studies all showed that CAT can substantially reduce the number of administered items, while maintaining a high correlation between CAT latent trait (y estimates) and y estimates that were based on the entire item pool. Additionally, the studies by Hol et al (2001Hol et al ( , 2005 and MacDonald (2003) showed that reducing the number of administered items by CAT administration did not appreciably affect correlations with other criteria.…”

“…Furthermore, some studies have appeared involving real life CAT applications in attitude or personality testing using polytomous items 1 (Baek, 1995;Balasubramanian & Kamakura Wagner, 1989;Hol, Vorst, & Mellenbergh, 2005;Koch, Dodd, & Fitzpatrick, 1990;MacDonald, 2003). In the present article, the authors studied a CAT using polytomous items, as it is this use of CAT (i.e., polytomous items used in the measurement of personality and attitudes) that requires further study.…”

Computerized Adaptive Testing with the Zinnes and Griggs Pairwise Preference Ideal Point Model

“…Computerized adaptive testing (CAT) based on item response theory (IRT) models seems like a natural solution, and there have been several applications of CAT technology in the attitudinal and personality domains. Some of these applications have utilized dominance IRT models (e.g., Hol, Vorst, & Mellenbergh, 2005;Simms & Clark, 2005;Waller & Reise, 1989), whereas others have been based on ideal point IRT models (e.g., Roberts, Lin, & Laughlin, 2001). Yet the common thread here is that items have consisted of individual statements requiring respondents to indicate their level of agreement using a dichotomous or polytomous response format.…”

Ventajas de los Modelos Politómicos de Teoría de Respuesta al ítem en la Medición de Actitudes Sociales: El Análisis de un Caso

“…3. La demostración de las ventajas de los modelos politómicos para el diseño TAIs e¿ cientes (Hol, Vorst & Mellenbergh, 2005;Lai, Cella, Chang, Bode & Heinemann, 2003;Van Rijn, Eggen, Hemker & Sanders, 2002), para la medición del DIF (Kim, Cohen, Alagoz & Kim, 2007) y para la igualación de puntuaciones (Lee, Kolen, Frisbie & Ankenmann, 2001). 4.…”