Not Read, but Nevertheless Solved? Three Experiments on PIRLS Multiple Choice Reading Comprehension Test Items

Abstract

Multiple-choice (MC) reading comprehension test items comprise three components: text passage, questions about the text, and MC answers. The construct validity of this format has been repeatedly criticized. In three between-subjects experiments, fourth graders (N 1 D 230, N 2 D 340, N 3 D 194) worked on three versions of MC items from the Progress in International Reading Literacy Study 2001 reading comprehension test with relevant components successively deleted: "original version" (text, questions, MC-answers), "version without text" (questions, MC-answers), "version without text and without questions" (only MC-answers). Answering correctly the MC items became more difficult as the relevant information was eliminated. In the two narrative fictional texts presented, the students' performance of the version without text was not better than chance. Conversely in the informational (fictional) text, the students' performance of the version without text was better than chance. In the third condition, students' performance was never better than chance.

Reliability and Validity of PIRLS and TIMSS

“…Neither the inherent limitations of MC items (e.g., MC is not suited for the assessment and discussion of opinions; cf. Haladyna & Rodriguez, 2013) nor possible distractor-related biases (Sparfeldt et al, 2012) seem to affect the corresponding aspects of a valid competence assessment.…”

“…For example, in TIMSS 2007 (4th grade), most items stand alone (151 out of 179). This could be relevant, because the questions and the set of choices in reading items seem to provide contextual information that sometimes facilitates finding the correct option (e.g., Sparfeldt, Kimmel, Löwenkamp, Steingräber, & Rost, 2012).…”

Ein Plädoyer zur Qualitätssicherung schriftlicher Prüfungen im Psychologiestudium

“…B. Lindner et al, 2015;Sparfeldt, Kimmel, Löwenkamp, Steingräber & Rost, 2012). Wenn Prüfende nicht die zeitlichen Kapazitäten haben, um sicherzustellen, dass hochwertige und weitgehend fehlerfreie geschlossene Aufgaben erstellt werden können, sollte auf den Einsatz geschlossener Aufgaben verzichtet werden.…”

Experimental Test Validation

“…Albeit generally rare, this approach was sometimes chosen in the domain of reading comprehension (e.g., Katz, Lautenschläger, Blackburn, & Harris, 1990; Rost & Sparfeldt, 2007; Schroeder & Tiffin-Richards, 2014; Sparfeldt, Kimmel, Löwenkamp, Steingräber, & Rost, 2012). These authors found that – across several multiple-choice reading comprehension tests – omitting the text still resulted in correct responses above chance level.…”