Manipulating Processing Difficulty of Reading Comprehension Questions: The Feasibility of Verbal Item Generation

Abstract

Based on a previously validated cognitive processing model of reading comprehension, this study experimentally examines potential generative components of text‐based multiple‐choice reading comprehension test questions. Previous research (Embretson & Wetzel, 1987; Gorin & Embretson, 2005; Sheehan & Ginther, 2001) shows text encoding and decision processes account for significant proportions of variance in item difficulties. In the current study, Linear Logistic Latent Trait Model (LLTM; Fischer, 1973) parameter estimates of experimentally manipulated items are examined to further verify the impact of encoding and decision processes on item difficulty. Results show that manipulation of some passage features, such as increased use of negative wording, significantly increases item difficulty in some cases, whereas others, such as altering the order of information presentation in a passage, did not significantly affect item difficulty, but did affect reaction time. These results suggest that reliable changes in difficulty and response time through algorithmic manipulation of certain task features is feasible. However, non‐significant results for several manipulations highlight potential challenges to item generation in establishing direct links between theoretically relevant item features and individual item processing. Further examination of these relationships will be informative to item writers as well as test developers interested in the feasibility of item generation as an assessment tool.

The Construct Validity of a Reading Test Based on Narrative Texts

“…Yet, research (Sheehan & Ginther, 2001;Kintsch, 1994;Gorin, 2005) suggests that the sequencing of information in both the text and test questions is crucial to the validity of narrative assessment as examinees generally perform poorly on most test items that do not follow the story line of the text (Gorin, 2005), and the mismatch of the order of information in the text and test items can impact on the construction of coherent text representation and increase the difficulty of test questions (Kintsch, 1994;Gorin, 2005).…”

“…However, studies that have adopted Kintsch and van Dijk (1978)'s model of text comprehension (Sheehan & Ginther, 2001; Shin, 2002;Kintsch, 1994;Gorin, 2005) have highlighted the sequencing of information in both the text and test questions as crucial to narrative assessment. Sheehan and Ginther (2001) reported that the location of relevant information within a narrative was associated with the difficulty of MC questions on the TOEFL-2000 reading test.…”

“…Shin (2002) found that students poorly performed on most test items that could not follow the story line of the narrative as such questions did not call for operations examinees were likely to use to construct narrative macrostructure. On their part, Kintsch (1994) and Gorin (2005) reported that changing the order in which the information in the narrative is presented impacted on the construction of coherent narrative representation, and this was found to increase the difficulty of test questions pertaining to the newly ordered information.…”

“…Although research insights suggest that the assessment of narrative comprehension should focus on both the microstructure and macrostructure abilities (Gagarina, et al, 2012;), and that it should be sensitive to the ordering of information in both the text and the test tasks (Sheehan & Ginther, 2001;Kintsch, 1994;Gorin, 2005) so as to represent the underlying narrative discourse competence, there is an abject paucity of literature that incorporates these three aspects within a single validation framework. This study therefore aims to fill this gap.…”

Mise sur pied d’une mesure de la compréhension de textes en langue seconde

“…Au fil du temps, les techniques pour évaluer la compréhension se sont améliorées et diversifiées de façon à refléter le mieux possible la compréhension chez le lecteur. Par exemple, avec la technique des questions à choix multiples (QCM), puisque des leurres aberrants seraient souvent écartés sans même qu'une véritable lecture du texte ait été faite, il est souhaitable de proposer des leurres plausibles (Tagliante, 2005 l'impact possible de la formulation des questions -rédigées, par exemple, à la négative -sur la compréhension du lecteur (Gorin, 2005).…”