Evaluating an Underdetermined Response Type for the Computerized Sat

Abstract

We evaluated a machine-scorable, computer-delivered response type for measuring quantitative reasoning skill. "Generating Examples" (GE) is built around items that present constraints and ask candidates to give one or more answers that meet those constraints. These items are attractive because, like many real-world problems, GE items can have multiple correct answers. In addition, they appear to tap cognitive processes somewhat distinct from those measured by conventional quantitative questions. Nine GE forms were spiraled among a sample of academically precocious youth taking the Computerized SAT in association with a national talent search program. The forms differed in item manipulations designed to affect difficulty and in the expected time per item needed for solution. Results showed that across item lengths, the insertion of certain constraints increased difficulty. In addition, after correcting for attenuation, GE items similar in time requirements to SAT Mathematical items correlated in the mid-eighties to midnineties with SAT Mathematical scores, indicating that GE items might fit reasonably well with the SAT. key words: constructed response, mathematical reasoning, new item types Evaluating an Underdetermined Response Type for the Computerized SAT Large-scale assessment programs have typically measured mathematical skill using multiple-choice items having a single best answer. This format is used because it can be machine scored. However, relying solely on singlebest-answer questions arguably narrows construct representation because problems with multiple acceptable answers are found in most criterion situations and may require different solution processes (Frederiksen~1984).Generating Examples (GE) problems require the examinee to give responses that meet a specified set of constraints. In contrast to traditional measures, GE problems are typically underdetermined: not enough information is given to identify the answer uniquely. These problems, as a result, can have many--possibly an infinite number of--correct answers. (A simple instance would be: "Give three numbers for which the mean is at least twice the median.") A framework for understanding how GE and more conventional welldetermined problems differ has been proposed by Nhouyvanisvong, Katz, and Singley (1997). Based on cognitive analysis of examinee solution procedures, these investigators concluded that the strategies test candidates use to solve GE problems differ from the formal algebraic methods typically employed to attack well-determined (single-solution) items. In particular, GE problems require the use of a generate-and-test approach because they cannot be solved purely through manipulating algebraic equations. In a generateand-test approach, the examinee proposes a solution to the problem, then checks whether that solution meets the problem constraints. Generate-andtest (also called "guess-and-check") is frequently observed in solving conventional problems (e.g., Katz, Friedman, Berinett, & Berger, 1996). However, because GE p...

Factors Affecting Difficulty in the Generating Examples Item Type

“…However, as noted in previous work (Nhouyvanisvong & Katz, 1998;Bennett et al, 1999a), not all constraints increase the amount of testing. Nhouyvanisvong and Katz distinguish between generator and verifier constraints.…”

“…Two previous studies of GE items provide support for the predictions that (a) adding a verifier constraint to an item will increase difficulty, whereas (b) an additional generator constraint will not. Nhouyvanisvong and Katz (1998) investigated GE items designed at the level of the quantitative section of the Graduate Record Examinations (GRE ® ) General Test, whereas Bennett et al (1999a) investigated the use of GE items in the context of the mathematics portion of the SAT ® I: Reasoning…”

“…The predictive success of solution density is especially noteworthy, because a previous analysis of GE items (Bennett et al, 1999a) failed to find a relationship between density and performance.…”

“…First, Bennett et al (1999a) conducted their analysis of density post hoc: The items were not specifically crafted to manipulate density, and in fact, density and constraint type (generator or verifier)…”

“…These constraints do not determine a unique solution, so GE items can have multiple correct solutions. Previous research on this item type has investigated the types of skills it elicits -compared with the types of skills elicited by more standard algebra items -both through cognitive (Nhouyvanisvong, Katz, & Singley, 1997) and psychometric analyses (Bennett, Morley, Quardt, Rock, & Katz, 1999a;Bennett et al, 1999b). Investigations of item factors that affect the difficulty of GE items have focused largely on simple structural characteristics (e.g., number of solutions requested or instructions on the solvability of the items).…”