Information Structure for Geometric Analogies: A Test Theory Approach

Abstract

Although geometric analogies are popular items for measuring intelligence, the information processes that are involved in their solution have not been studied in a test theory context. In the current study, processing is examined by testing alternative models of information structure on geometric analogies. In contrast to the treatment of models in other studies that have appeared in the cognitive literature, the models are tested jointly as mathematical models of processing and as latent trait models of individual differences. The joint model-ing was achieved by applying the one-parameter linear logistic latent trait model to predict response accuracy from information structure. The results supported the model that distinguished between spatial distortion and spatial displacement transformations, which have opposite effects on item difficulty. Further, no significant sex difference in overall accuracy or processing were observed. Implications of the results for processing mechanisms and test design are discussed. Geometric analogies are popular items for measuring nonverbal intelligence. They are included on tests such as the Cognitive Abilities Test (CAT), the Lorge-Thorndike Intelligence Test, and the American Council on Education Examination (ACE). An example of a geometric analogy that is similar to CAT items is given in Figure 1, along with structural notation. The task is to select from the alternatives (D1 to Ðs) to find a figure that is related to C in the same way that B is related to A.Similar to most types of intelligence test items, validation studies on geometric analogies have been mainly concerned with trait organization rather than the nature of processing involved in item performance. That is, person scores are correlated with other tests or achievement and learning, but studies of processing are infrequently reported. Thus, the psychometric validation research has not clarified the nature of the reasoning process measured by geometric analogies.A new type of validity research, cognitive component analysis of aptitude (e.g., Pellegrino & Glaser, 1979; Sternberg, 1977a; Whitely, in press), seeks to understand test validity by identifying the information processes that contribute to performance. Cognitive component analysis has been applied to several types of aptitude test items, including verbal analogies (

Generating items during testing: Psychometric issues and models

“…First, item properties are often predicted at least moderately well. Studies from my laboratory have concerned ten different item types, including verbal analogies and verbal classifications (Embretson, 1984;Embretson, 1985;Embretson & Schneider, 1989;Embretson, Schneider & Roth, 1985;Whitely, 19801), geometric analogies and geometric classifications (Whitely & Schneider, 1981), series completion (Embretson, 1985), paragraph comprehension (Embretson & Wetzel, 1987), spatial folding (Embretson, 1991(Embretson, , 1994, mathematics word problems (Embretson, 1995a), matrix completion problems (Embretson, 1995c; and, most recently, spatial construction items. Prediction levels comparable to a multiple correlation of at least .70 are usually obtained.…”

Cognitive Diagnostic Analysis of Reading Comprehension Items: The Case of English Proficiency Assessment in Saudi Arabia

“…The results are provided in Table 4. Using the MAD cutoff values for levels of ICC recovery described earlier, (a) there are two items (11 and 29) with very good recovery, MAD<0.02, (b) 13 items (3,6,7,10,14,16,17,21,24,27,32,35,39) with good recovery, 0.02 ≤ MAD<0.05, (c) seven items (12,13,18,26,28,36,37) with somewhat good recovery, 0.05 ≤ MAD<0.07), (d) six items (2,15,19,22,25,34) with somewhat poor recovery, 0.07 ≤ MAD<0.10, (e) and 11 items (1,4,5,8,9,20,23,30,31,33,38) The presence of items with unsatisfactory (somewhat poor or poor) ICC recovery is not a surprise because it is not realistic to expect that the small number of attributes with relatively high level of generality, used in this study, would be sufficient to fully explain the conditional probabilities of correct response for all test items. Nevertheless, the ICC fit and misfit of STEP-RC items provide information in line with the purpose of the present study; (more details on that matter are provided in the discussion part).…”

“…CDMs have been developed in a variety of frameworks such as (a) item response theory (IRT), [17][18][19], (b) rule space methodology [14,[20][21][22], (c) latent class modeling [6,9,[23][24][25], (d) IRT-based latent class modeling [26][27][28], and (e) Bayesian modeling [29]. Although practical applications of CDMs exist in different areas of assessment, such as mathematics [30][31][32][33], language testing [8,[34][35][36][37], and psychology [12,16], the current practice is still behind theoretical and methodological developments in the field of cognitive diagnosis modeling. Particularly useful, yet underrepresented in published research, are CDM studies that (a) examine attributes which have been targeted in the process of developing specific large-scale assessment tools and (b) take into account the context of assessment and align the results with the scale of scoring and psychometric analysis of the specific tool.…”

Modelización de ítems de matrices figurales y pautas específicas propuestas para su construcción

“…Las reglas son operaciones cognitivas elementales que se requieren para resolver el ítem (Kubinger, 2008). Cada regla constituye "un determinado tipo de relación análoga" (Blum, Abal, Lozzia et al, 2011, p. 138), siendo ejemplos de reglas la adición (Arendasy, 2005;Freund, Hofer, & Holling, 2008), la sustracción (Arendasy, 2005), el desplazamiento espacial (e.g., rotación) y distorsiones del tamaño y de la forma (Whitely & Schneider, 1981). Es posible crear reactivos con diferentes reglas o combinaciones de reglas (Kubinger, 2008;Zeuch, 2010).…”