Three Response Types for Broadening the Conception of Mathematical Problem Solving in Computerized Tests

Abstract

Three open-ended response types-mathematical expression (ME), generating examples (GE), and graphical modeling (GM)-are described that could broaden the conception of mathematical problem solving used in computerized admissions tests. ME presents single-best-answer problems that call for an algebraic formalism, the correct rendition of which can take an infinite number of surface forms. GE presents loosely structured problems that can have many good answers taking the form of a value, letter pattern, expression, equation, or list. GM asks the examinee to represent a given situation by plotting points on a grid; these items can have a single best answer or multiple correct answers. For the three basic types, sample items are provided, the examinee interfaces and approaches to automated scoring are described, and research results are reported. It is illustrated how ME, GE, and GM can be combined to form extended constructed-response problems, and a description is offered of how item classes might be used as a basis for creating production-ready scoring keys. Index terms: automated scoring, computerbased testing, constructed response, mathematics performance assessment.The traditional paper-and-pencil (P&P), multiple-choice (MC) item format consists of a static stimulus followed by a series of response options. This format has served testing programs well for many years because its compactness allows for great breadth of coverage-many items can be administered in a short period. It is also cost efficient because it can be machine scored.However, this traditional format cannot effectively measure some constructs; in particular, when the target construct requires either a dynamic stimulus (e.g., listening comprehension) or a complex response (e.g., writing an essay, composing a computer program, producing a building design). To handle constructs requiring dynamic stimuli, large-scale testing programs have typically combined P&P with video or audio tape, producing a serviceable but expensive and administratively cumbersome assessment. To accommodate constructs calling for complex responses, testing programs have increasingly employed performance tasks, which also can be uneconomical due to the need for human scoring.Computerized testing has brought with it the potential for "new" assessment tasks. Some of these tasks might be more efficiently delivered versions of tasks used in traditional testing programs; others might be truly new, in that they measure constructs that could not be measured by P&P MC tests.These new tasks can be divided into three classes. In the first class, the stimulus is dynamic. An operational example is the Listening section of the Test of English as a Foreign Language (Educational Testing Service, 1999). Digitally recorded audio and context-setting photos are presented, followed by MC questions. One advantage of this digital presentation is the consistency in quality with which the same audio stimulus can be delivered from one examinee to the next.

Using G-theory in the development of performance assessment of the socio-emotional domain of children

“…Computerized scoring procedures for performance assessment are currently receiving considerable attention (Bennett, 1999;Bennett et al, 2000;Brennan, 2000). This study is a descriptive research in order to collect feedback from respondents as well as to survey error resources in measurement.…”

A Historical Survey of Research Regarding Constructed-Response Formats

“…Randy Bennett took the lead at ETS in exploring the technology for scoring constructed responses in concert with theory about the relevant constructs, including mathematics Bennett et al 1999Bennett et al , 2000aSandene et al 2005;Sebrechts et al 1991Sebrechts et al , 1996, computer science (Bennett and Wadkins 1995), graphical items (Bennett et al 2000a;, and formulating hypotheses (Bennett and Rock 1995). The scoring of mathematics items has reached a significant level of maturity (Fife 2013), as has the integration of task design and automated scoring (Graf and Fife 2012).…”

“…He evaluated the SAT grid-in format with GRE items and concluded that the multiple-choice and grid-in versions of GRE items behaved very similarly. Following the adoption of the grid-in format in the SAT, a more comprehensive examination of mathematics item formats that could serve to elicit quantitative skills was undertaken, informed by advances in the understanding of mathematical cognition and a maturing computerbased infrastructure (Bennett and Sebrechts 1997;Bennett et al , 1999Bennett et al , 2000aSandene et al 2005;Sebrechts et al 1996). More recently, the mathematics strand of the CBAL initiative has attempted to unpack mathematical proficiency by means of competency models, the corresponding constructed-response tasks (Graf 2009), and scoring approaches (Fife 2013).…”

Advancing Human Assessment: A Synthesis Over Seven Decades

“…Streams of research on essay scoring and short-text scoring expanded (Attali and Burstein 2006;Leacock and Chodorow 2003;Powers et al 2002;Quinlan et al 2009), a new line on speech scoring was added (Zechner et al 2007(Zechner et al , 2009, and publications were released on the grading of graphs and mathematical expressions (Bennett et al 2000).…”