Answering the New SAT Reading Comprehension Questions Without the Passages

Abstract

It has been reasonably well established that test takers can, to varying degrees, answer some reading comprehension questions without reading the passages on which the questions are based, even for carefully constructed measures like the Scholastic Aptitude Test (SAT). The aim of this study was to determine what test-taking strategies examinees use, and which are related to test performance, when reading passages are not available. The research focused on reading comprehension questions similar to those that will be used in the revised SAT, to be introduced in 1994. The most often cited strategies involved choosing answers on the basis of consistency with other questions and reconstructing the main theme of a missing passage from all of the questions and answers in a set. These strategies were more likely to result in successful performance on individual test items than were any of many other possible (and less constructrelevant) strategies.

Using working memory theory to investigate the construct validity of multiple-choice reading comprehension tests such as the SAT.

“…On the basis of previous findings (e.g., Katz et al, 1990;Powers & Leung, 1995), we predicted that mean performance would be above chance (20%) even when test takers could not read or consult the passages on which the questions were based. However, if we were correct in concluding that test takers were still doing a significant amount of passage reading even though they were not allowed to do any advance passage reading in the question-driven strategy in Experiment 2, then we would expect performance without any passage reading to be significantly lower than 63%, which was the mean performance Jevel for the no advance reading strategy in Experiment 2.…”

“…Although the primary goal of the present research was to investigate strategies that students might plausibly be using in the conventional SAT setting, our final experiment included a condition in which students were required to answer the SAT questions without consulting the passages at all (see also Katz et al, 1990;Powers & Leung, 1995). By comparing the relative power of reading span and the Nelson-Denny at predicting SAT performance when test takers do or do not see the passages, we were able to provide support for our view that the SAT assesses passage comprehension as long as the test taker is allowed to do some passage reading.…”

“…A number of recent studies have shown that spatial process plus storage measures do not predict comprehension ability (see, e.g., Shah & Miyake, 1996). Indeed, Katz et al's claim was based on their finding that college students could perform at better-than-chance levels on the multiple-choice reading questions on pre-1994 versions of the SAT even though they were not given access to the passages (see also Powers & Leung, 1995, for a similar result with the revised version of the SAT). Given that the Educational Testing Service (ETS) developed the SAT as a measure of the ability to read and comprehend short English prose passages (Donlon, 1984), Katz et al reasoned that test takers "must answer a group of multiplechoice items based on what is stated or implied in a passage, and not choose answers on the basis of personal opinion or prior knowledge, or because a particular choice is known to be true" (p. 122).…”

“…However, we believe that these kinds of strategies play a minor role when test takers try hard to answer questions without the help of the passages on which they are based. Self-reports provided by students taking the SAT test without the passages (Powers & Leung, 1995) strongly suggest that the better-than-chance performance is a result of strategies that "entail legitimate verbal reasoning abilities" (Powers & Leung, 1995, p. 106), strategies such as "choosing answers on the basis of consistency with other questions and reconstructing the main theme of a missing passage from all the questions and answers in the set" (Powers & Leung, 1995, p. 105). These kinds of sophisticated verbal reasoning strategies are precisely the kinds of strategies that have been shown to draw on the processing and storage resources of working memory (Jurden, 1995;Kyllonen & Christal, 1990).…”

“…As Powers and Leung (1995) pointed out, the no-passage strategy "is neither efficient nor effective" (p. 125). Powers and Leung found that students used nearly as much time answering the questions without reading the passages as is allowed on the exam for reading the SAT passages and answering the questions.…”