The Prediction of Toefl Reading Comprehension Item Difficulty for Expository Prose Passages for Three Item Types: Main Idea, Inference, and Supporting Idea Items

Abstract

The purpose of the current study is to predict the difficulty (equated delta) of a large sample (n=21.3) of TOEFL reading comprehension items. (Only main idea, inference, and supporting statement items were sampled.) A related purpose was to examine whether text and text‐related variables play a significant role in predicting item difficulty; we argued that evidence favoring construct validity would require significant contributions from these particular predictor variables. In addition, details of item predictability were explored by evaluating two hypotheses: (1) that multiple‐choice reading comprehension tests are sensitive to many sentential and discourse variables found to influence comprehension processes in the experimental literature, and (2) that many of the variables identified in the first hypothesis contribute significant independent variance in predicting item difficulty. The great majority of sentential and discourse variables identified in our review of the experimental literature were found to be significantly related to item difficulty within TOEFL's multiple‐choice format. Furthermore, contrary to predictions which we attributed to critics of multiple‐choice tests, the pattern of correlational results showed that there is a significant relationship between item difficulty and the text and text‐related variables. We took this as evidence supporting our claim that multiple–choice reading items yield construct valid measures of comprehension. That is, since critics have pointed out that reading items can often be correctly answered without reading of the text passage, this seems to imply that item variables (not text nor text‐related variables) should fbe prominent predictors of reading item difficulty. Since the contrary relationship was found, we concluded that this provides evidence favoring construct validity. We found, further, in several stepwise linear regression analyses, that many of these text and taxt‐related variables provide independent contributions in predicting reading item difficulty. This was interpreted as providing additional support for construct validity. More specifically, apart from the correlational results, the following stepwise linear regressions results were obtained. For the full sample of 213 items, and where equated delta (an index of item difficulty) is the dependent variable, we found 33 percent (p < .0001) of the variance of item difficulty could be accounted for by sight variables. All sight variables reflected significant and independent contributions due solely to text and text/item overlap variables. This result provided evidence favoring constract validity of the TOEFL reading comprehension items. We also conducted a separate analysis of a subset (n=98) of the full set of 213 items to examine the possible statistical effect of nesting in the original sample, (Nesting occurs when several items relating to the same passage are analysed together; a non‐nested subset is formed when only one item per passage is used.) Eleven variables accounted for 58 per...

An Analysis of Factors Affecting the Difficulty of Dialogue Items in Toefl Listening Comprehension

“…The length of stimulus and the amount of information the examinee needs to process to determine the correct answer are among the variables that contributed to item difficulty in cloze items (Brown, 1988; Abraham & Chapelle, 1993). Freedle and Kostin (1993) found that the presence of negation contributed significantly to the difficulty of reading comprehension items. However, these variables and others have not been examined in the context of multiple-choice listening items.…”

“…Numerous researchers have examined factors affecting the difficulty of various item types (e.g., Brown, 1988; Abraham & Chapelle, 1992;Freedle & Kostin, 1993). The length of stimulus and the amount of information the examinee needs to process to determine the correct answer are among the variables that contributed to item difficulty in cloze items (Brown, 1988; Abraham & Chapelle, 1993).…”

“…Freedle and Kostin (1993) reported that the number of negatives in an item contributes to item difficulty in reading comprehension items. In this study, negatives were analyzed separately in the stimulus, the correct answer, and the distractors.…”

Using Necessary Information to Identify Item Dependence in Passage-Based Reading Comprehension Tests

“…Studies of item difficulty have shown that properties of the necessary information, such as location, vocabulary difficulty, propositional density, and correspondence to the key and distractors, are stronger predictors of item statistics than characteristics of the passage as a whole (Embretson & Wetzel, 1987;Freedle & Kostin, 1993;Sheehan & Ginther, 2000;Sheehan, Kostin, & Persky, 2006). That is, the passage itself is less important than the portion of the passage needed to answer the question.…”

A Tree‐based Approach to Proficiency Scaling

“…The class of "Items requiring comparisons" was operationally defined to include all items in which the solution process involved drawing a comparison between two or more concepts, ideas, stylistic techniques, viewpoints, solutions, arguments or causes. Following Freedle and Kostin (1991), the set of items requiring use of a negation process was defined to include all items containing actual negations ("no", "not"), as well as all items containing negated words Cuseless", "Ynknown").…”