The Effect of Question Order on Responses to Multiple-choice Questions

Abstract

Educators and researchers often make the assumption that the order of test or survey questions is unimportant. Is this assumption valid? This study investigates how the order of two related FCI questions (#13 and 14) affects students' responses. This study also investigates the effect an unrelated FCI question (#23) has on answers to the above problems. Four versions of a survey were administered before and after instruction to 243 students taking an algebra-based physics class. Versions 1 and 2 of the survey included the related physics questions in opposite order. Versions 3 and 4 included the unrelated physics question and one of the above questions. Student responses for the four versions were compared for both the pre-and post-instruction surveys. IntroductionFor years physics education researchers have used surveys and other investigative tools to learn what physics students know and how they think. These researchers often assume that the order of questions does not affect student responses. For as long as teachers have been giving tests, students have been looking for strategies and "tricks" to improve their performance on exams. These methods include "test taking strategies" often taught in high school and some college orientation classes. A common strategy includes looking for the answer or helpful hints in other problems when answering an unfamiliar question.Thus, students' performance may be influenced by their test taking skills or ability to "learn" the physics as they take the test. There is no reason to believe that students forget this skill when they are taking physics education research surveys as opposed to graded exams. This is a serious problem for researchers who are attempting to understand how students think about a series of problems or situations.Educational researchers have for years studied the effect of question order, beginning with Mollenkopf [1] and others in the 1950s. Most early studies [2] focused on changing the order of the ease of the questions: "easy to hard" and "hard to easy." With the advent of computerized adaptive testing educational researchers have begun to look more closely at the effect of changing the context and order in which test items appear. [3] The purpose of this paper is to test the aforementioned assumption that the order of questions in a test does not affect student responses to individual questions on the test. The results of our study described here suggest that the order of questions does

“On Second Thoughts…”: Changes of Mind as an Indication of Competing Knowledge Structures

“…By exploring how our student cohorts answered FCI Q15 in the presence or absence of FCI Q16 and through classroom observation of student interactions and discussion, there appears to be a strong interaction between these two questions, consistent with Gray et al 21 and Gray. 22 This paper investigates that interaction and, using the resources model as a theoretical framework, explores what the interaction between the questions may be telling us about the knowledge structures held by our students and the conditions under which those structures are being activated.…”

“…Gray et al 21 and Gray, 22 using the 1992 version of the FCI, 12 found that swapping the order in which the 1992 equivalents of Q15 and Q16 were presented to students had a small effect on their ability to correctly answer both the questions. When they were presented in reverse order as a two-question test, slightly fewer students chose the correct answer for Q15 than when the questions were presented in the usual order, and slightly more students chose the correct answer for Q16.…”

“…22 This paper investigates that interaction and, using the resources model as a theoretical framework, explores what the interaction between the questions may be telling us about the knowledge structures held by our students and the conditions under which those structures are being activated. We explore the effect of cueing on student responses and show how students can transition rapidly between correct and incorrect answers-and between sound and fragile reasoning-based not just on how a particular question is asked but also on the context and sequencing of the questions.…”

Transfer Between Paired Problems In An Interview

“…[6] In the transcript discussed above, the spaceship question caused the student to switch from one wrong answer to another wrong answer. While previous transfer models would see previous experience or knowledge learned in class as the key source of transferred information, looking at the transcript from the actor-oriented transfer perspective, it becomes clear that the main source of information can be the problems themselves.…”

“…The following interview was done as part of a larger project unrelated to transfer [6], yet reanalyzing it from the perspective of transfer sheds new light onto the students' understanding and on the usefulness and importance of Lobato's actor-oriented transfer model. Since transfer is defined by the student, often the concepts transferred by students are not correct physics concepts.…”