Changing the Order of Mathematics Test Items: Helping or Hindering Student Performance?

Abstract

This paper recounts an experiment by a mathematics professor who primarily teaches mathematics majors. The main question explored is whether the ordering of the questions makes a difference as to how students perform in a test. More specifically we focus here on the following research questions: (1) Does arranging a math test with easy-to-hard items versus hard-to-easy items impact student performance? and (2) If so, does item order impact male and female mathematics majors and non-majors in unique ways? We examine data collected over multiple semesters with several different classes. We find that for most of the mathematics students who were examined, the ordering of the questions on a test did not impact performance. However, female majors performed better on classroom exams when the test was arranged with the more difficult questions presented first. Readers who are interested in teaching mathematics, educational psychology, or gender issues in the classroom may find our results intriguing.Evolution of a research project. A mathematics professor, I (Kristin Kennedy) often give two versions of the same exam to reduce the possibility of students copying from one another. Recently, I began to wonder if the order of the questions could affect a student's grade. I approached Allison Butler, professor of psychology, for her expertise. I of course wanted to ensure that both tests were fair, and Professor Butler was interested to see if there was any difference from a psychological point of view. This paper describes our joint inquiry into this matter.

A review of relationships between item sequence and performance on multiple-choice exams.

“…Before researchers can sequence items by difficulty level, they need a systematic way of judging item difficulty. One approach for judging difficulty level is to assume that items requiring multiple steps and application of content would be more difficult than one-step questions requiring only factual answers (Kennedy & Butler, 2013). In contrast, basing judgments of item difficulty on the percentage of students missing particular items appears to be the most parsimonious and direct way for judging item difficulty.…”

“…Unless instructors have access to highly sophisticated technology that individualizes item sequence according to a student’s current response pattern, instructors might choose to go with an item sequence most generally supported in the research literature. Additionally, research comparing the impact of different item sequences on exam scores has not generally yielded significant results (Huck & Bowers, 1972; Kennedy & Butler, 2013; Vander Schee, 2013; Weinstein & Roediger, 2010, 2012).…”

“…Studies on the relationship between the items sequenced by difficulty and students’ performance on tests conducted from the 1950s through the 1980s and then from 2010 and later lack consistency in results. Relatively few previous studies have shown test performance to be better when items are sequenced from easy to hard (e.g., Sax & Cromack, 1966; Hambleton & Traub, 1974; Hodson, 1984; Plake et al, 1982), and even fewer studies have shown superior test performance with hard to easy sequencing of exam items (Kennedy & Butler, 2013).…”

“…The previous section highlighted the results found in the studies comparing different conceptual themes and difficulty levels in grouping exam items. This section reports the methodological specifics for assessing the difficulty level of items in several of these studies (Barcikowsky & Olsen, 1975; Hambleton & Traub, 1974; Kennedy & Butler, 2013).…”

“…Among the many factors that could affect performance on multiple-choice tests is the sequence of test items. Some research (Hambleton & Traub, 1974; Kennedy & Butler, 2013; Plake, Ansorge, Parker, & Lowry, 1982) indicates that when the content of items is held constant, the order of those items can affect performance on the tests. However, a far larger number of studies reviewed in the current analysis have questioned the impact of item sequence on exam performance.…”