Passage-Dependence of Items Designed to Measure the Ability to Identify the Main Ideas of Paragraphs: Implications for Validity

Abstract

Items from a standardized reading test designed to measure the ability to identify the main ideas of paragraphs were administered without the associated paragraphs. The two samples consisted of graduate students and inner-city high-school students. A random half of each sample was given brief, general directions for answering the items in the absence of the passages, and the other half was given more extensive directions for answering the items in the absence of the passages. The different directions did not have a significant effect on performance, but both halves of the two samples answered a substantial number of items correctly. An index of passage-dependence was computed for each item, and the index values obtained from the responses of the graduate students and high-school students were substantially correlated. READING comprehension is often measured by asking questions about specific passages that are included in the test. Various writers have noted that unless the passages and items are carefully written, it may be possible for examinees to answer correctly the questions without paying careful attention to the associated passages (e.g., Davis, 1964, p. 105). This contention has been supported, in part, by Preston (1962), Weaver andBickley (1967), andPyrczak (1972), all of whom administered standardized readingcomprehension items without the passages. Pyrczak, furthermore, investigated the sources of information and misinformation that examinees used in responding to questions when they did not have access to the passages. In that study, it appeared that examinees at UNIV OF MONTANA on April 4, 2015 epm.sagepub.com Downloaded from 344 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT relied upon their general information (or misinformation) as well as the interrelationships among the items on a given passage to select choices. These two bases for responding could constitute a serious threat to the validity of reading-comprehension tests. The present study is concerned with the passage-dependence of readingcomprehension items in response to which neither of these bases for responding would be helpful.Specifically, items in Part II of the Traxler High School Reading Test (Form A) were used in this investigation. Each of the items was designed to measure the ability to find the main idea in a given paragraph. Most of the distracters in these items appear to be true statements but are incorrect as choices because they are not the main idea of the associated paragraph. Thus, it does not seem likely that examinees can use their general knowledge of -the truth or falsity of each choice as a basis for identifying the keyed choices. Furthermore, since each of the items is based upon a separate paragraph, there are no apparent interrelationships among items that examinees can use as a basis for responding.Despite these apparent strengths of the items in the Traxler High School Reading Test, Tuinman (1972, pp. 49-50) suggested that &dquo;In many instances the right answer can be found without reading the passages at all.&d...

Answering the New SAT Reading Comprehension Questions Without the Passages

“…This finding is, of course, not a new discovery. It has long been acknowledged that reading comprehension questions vary in the ~degree to which they depend on the passages (Preston, 1962), and this phenomenon has frequently been studied (Preston, 1964;Pyrczak, 1972Pyrczak, , 1974Pyrczak, , 1975Tuinman, 1974;Weaver & Bickley, 1967). Nor is this finding surprising, for as Conlan (1990) argued, it would be highly unusual, given current conceptions of reading comprehension, if examinees were not able to extract some information from the test questions themselves.…”

The Relative Merits of Item Transformations and the Cloze Procedure for the Measurement of Reading Comprehension

“…Farr & Smith, 1970;Johns, 1978;Preston, 1964). This issue of the passage dependency of items is of paramount importance (Allington, Chodos, Domaracki, & Truex, 1977;Pyrczak, 1972Pyrczak, , 1974Pyrczak, , 1975Pyrczak, , 1975Pyrczak, -76, 1976Pyrczak & Axelrod, 1976;Tuinman, 1972Tuinman, -73, 1973Weaver, Bickley, & Ford, 1969).…”

Not Read, but Nevertheless Solved? Three Experiments on PIRLS Multiple Choice Reading Comprehension Test Items

“…Research has provided evidence for the so-called "passage-independence, that is, the likelihood that on some items, an examinee could respond correctly : : : without having read the accompanying passage" (Coleman, Lindstrom, Nelson, Lindstrom, & Gregg, 2010, p. 244). Consequentially, there is a long history of criticism on multiple-choice reading comprehension test items as well (e.g., Coleman et al, 2010;Drum et al, 1981;Katz, Blackburn, & Lautenschlager, 1991;Katz, Lautenschlager, Blackburn, & Harris, 1990;Katz, Marsh, Johnson, & Pohl, 2001;Keenan & Betjemann, 2006;Lifson, Scruggs, & Bennion, 1984;Powers & Leung, 1995;Preston, 1964;Pyrczak, 1972Pyrczak, , 1974Pyrczak, , 1976Rost & Sparfeldt, 2007;Samuels, 1968;Tian, 2006;Tuinman, 1973Tuinman, -1974. Katz et al (1990, Experiment 1), for example, administered 100 multiple-choice reading comprehension test items stemming from the verbal reasoning task of the Scholastic Aptitude Test (SAT) without presenting the corresponding text passages to college students.…”

“…Pyrczak (1974) reported a substantial amount of passage independence in multiple-choice questions related to the identification of the main idea of singular paragraphs in a sample of graduate students and in a sample of high school students. Coleman et al (2010) reported, as previously mentioned, that examinees answered above chance levels literal comprehension items as well as interpretative comprehension items.…”