Graphical Modeling: A New Response Type for Measuring the Qualitative Component of Mathematical Reasoning

Bennett, Randy Elliot; Morley, Mary; Quardt, Dennis; Rock, Donald A.

doi:10.1002/j.2333-8504.1999.tb01819.x

Cited by 7 publications

(8 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…CFA indicated that the FH tasks assessed a different dimension of reasoning than the typical GRE tasks. Another group of ETS researchers (Bennett, Morley, Quardt, & Rock, 2000) investigated the function of a graphical modeling (GM) item type for potential use in the computer-based GRE. The GM items required examinees to draw a graph modeling in a given situation.…”

mentioning

confidence: 99%

Measurement Properties of Two Innovative Item Formats in a Computer-Based Test

Wan

Henly

2012

Applied Measurement in Education

View full text Add to dashboard Cite

Many innovative item formats have been proposed over the past decade, but little empirical research has been conducted on their measurement properties. This study examines the reliability, efficiency, and construct validity of two innovative item formats-the figural response (FR) and constructed response (CR) formats used in a K-12 computerized science test. The item response theory (IRT) information function and confirmatory factor analysis (CFA) were employed to address the research questions. It was found that the FR items were similar to the multiple-choice (MC) items in providing information and efficiency, whereas the CR items provided noticeably more information than the MC items but tended to provide less information per minute. The CFA suggested that the innovative formats and the MC format measure similar constructs. Innovations in computerized item formats are reviewed, and the merits as well as challenges of implementing the innovative formats are discussed.Recent years have seen rapid growth in the use of computers in educational and licensure/certification testing. Many tests are now administered on computer, and software has been developed to facilitate all aspects of test development. On paper-and-pencil tests, multiple-choice (MC) items are the dominant testing format (Scalise & Gifford, 2008). Now armed with new technologies, test users and developers have shown increased interest in employing innovative item formats.Many innovative item formats have emerged over the past decade, but little empirical research has been conducted on their measurement properties. The purpose of this study is to empirically examine the reliability, efficiency, and Correspondence should be addressed to Lei Wan,

show abstract

mentioning

confidence: 99%

Measurement Properties of Two Innovative Item Formats in a Computer-Based Test

Wan

Henly

2012

Applied Measurement in Education

View full text Add to dashboard Cite

show abstract

“…The reactions to GM followed the same pattern obtained for other constructed-response items. Bennett et al (2000) found that more examinees preferred MC graphical reasoning questions to GM items (41% to 35%). However, when asked which was the fairer indicator of their ability to undertake graduate study, their preferences reversed (42% to 31%).…”

Section: Figurementioning

confidence: 96%

“…Reproduced by Permission.) Bennett et al (2000) provided data relevant to the potential for GM to produce adverse impact. Because GM is graphical, adverse impact is a reasonable concern, especially given the well-established finding that men outperform women on certain spatial tasks (Linn & Petersen, 1985).…”

Section: Figurementioning

confidence: 99%

“…The psychometric characteristics of GM were also investigated by Bennett et al (2000). GM was found to be highly reliable, with coefficient α values in the low .90s (for an 18-item test).…”

mentioning

confidence: 99%

“…That the slope of the equation is positive. Bennett, Morley, Quardt, & Rock (2000) designed two GM forms and created scoring keys from the problem information contained in the item stems. They administered the forms to 243 individuals planning to attend graduate school or enrolled as first-year graduate students.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Three Response Types for Broadening the Conception of Mathematical Problem Solving in Computerized Tests

Bennett¹,

Morley²,

Quardt

2000

Applied Psychological Measurement

Self Cite

View full text Add to dashboard Cite

Three open-ended response types-mathematical expression (ME), generating examples (GE), and graphical modeling (GM)-are described that could broaden the conception of mathematical problem solving used in computerized admissions tests. ME presents single-best-answer problems that call for an algebraic formalism, the correct rendition of which can take an infinite number of surface forms. GE presents loosely structured problems that can have many good answers taking the form of a value, letter pattern, expression, equation, or list. GM asks the examinee to represent a given situation by plotting points on a grid; these items can have a single best answer or multiple correct answers. For the three basic types, sample items are provided, the examinee interfaces and approaches to automated scoring are described, and research results are reported. It is illustrated how ME, GE, and GM can be combined to form extended constructed-response problems, and a description is offered of how item classes might be used as a basis for creating production-ready scoring keys. Index terms: automated scoring, computerbased testing, constructed response, mathematics performance assessment.The traditional paper-and-pencil (P&P), multiple-choice (MC) item format consists of a static stimulus followed by a series of response options. This format has served testing programs well for many years because its compactness allows for great breadth of coverage-many items can be administered in a short period. It is also cost efficient because it can be machine scored.However, this traditional format cannot effectively measure some constructs; in particular, when the target construct requires either a dynamic stimulus (e.g., listening comprehension) or a complex response (e.g., writing an essay, composing a computer program, producing a building design). To handle constructs requiring dynamic stimuli, large-scale testing programs have typically combined P&P with video or audio tape, producing a serviceable but expensive and administratively cumbersome assessment. To accommodate constructs calling for complex responses, testing programs have increasingly employed performance tasks, which also can be uneconomical due to the need for human scoring.Computerized testing has brought with it the potential for "new" assessment tasks. Some of these tasks might be more efficiently delivered versions of tasks used in traditional testing programs; others might be truly new, in that they measure constructs that could not be measured by P&P MC tests.These new tasks can be divided into three classes. In the first class, the stimulus is dynamic. An operational example is the Listening section of the Test of English as a Foreign Language (Educational Testing Service, 1999). Digitally recorded audio and context-setting photos are presented, followed by MC questions. One advantage of this digital presentation is the consistency in quality with which the same audio stimulus can be delivered from one examinee to the next.

show abstract

Objective testing of educational achievement.

Rodriguez¹,

Haladyna²

2013

APA Handbook of Testing and Assessment in Psychology, Vol. 1: Test Theory and Testing and Assessment in Industrial and Organiza

View full text Add to dashboard Cite

Educational achievement tests can be classified in many ways. Perhaps the most global classification is the distinction between standardized and teachermade tests. This distinction clarifies the degree to which content, format, administration, and scoring are standardized or left to the discretion of the teacher or testing agent (see Volume 3, Chapter 16, this handbook). Another distinction in educational tests is the format of the test items. More commonly used tests are classified as objectively or subjectively scored. The extent to which tests are objective or subjective is usually a function of the type of scoring required given the item or task type. These terms are components of a continuum rather than a dichotomous choice. Although some items might be considered strictly objectively scored because a correct response is known and responses may be machine scored (or scored with a key), other items may be subjectively scored because they require human scoring or a complex scoring guide such as a rubric, with which many correct (or partially correct) responses are possible. Even so, many forms of subjectively scored items, such as short-answer constructed-response (CR) items or even essays, may be scored with simple to complex computer scoring engines. Perhaps most performance assessments, with complex multidimensional aspects, are most unlikely to be objectively scored (see Chapter 20, this volume).When focused more specifically on the nature of the test items or tasks themselves, additional classification schemes are possible. The two categories used in this chapter are selected-response (SR) and CR items. For SR items, options are available from which to select responses, whereas for CR items, the respondent must construct or produce a response because none are available for selection. Similarly, items may be considered fixed-response items (SR) versus free-response or production items (CR). In practice, most educational test developers use the terms multiple choice (selected response) and constructed response because these formats are predominant in current educational tests. However, multiple choice as a label refers to a specific class of selection items, and so it is not all encompassing. Within the two classifications of SR items and CR items, there are many possible formats. In the class of CR items are formats that could be objectively scored, including the cloze, fill-in-the-blank, grid-in response, and short-answer formats and even some experiments or tasks that result in specific products. More subjectively scored production items include demonstrations, debates, essays, exhibitions, interviews, observations, and more complex performance tasks, portfolios, projects, and research papers (see Haladyna & Rodriguez, in press, for a comprehensive review of these types).The focus of this chapter is on objective testing of educational achievement and thus items that lend themselves to objective scoring (see Chapter 19, this volume, for additional perspectives). This category includes a wide range of SR i...

show abstract

Graphical Modeling: A New Response Type for Measuring the Qualitative Component of Mathematical Reasoning

Cited by 7 publications

References 8 publications

Measurement Properties of Two Innovative Item Formats in a Computer-Based Test

Measurement Properties of Two Innovative Item Formats in a Computer-Based Test

Three Response Types for Broadening the Conception of Mathematical Problem Solving in Computerized Tests

Objective testing of educational achievement.

Contact Info

Product

Resources

About