Modified scoring, traditional item analysis and Sato's caution index used to investigate the reading recall protocol

Abstract

The written recall protocol is increasingly being used in second language reading research as a measure of comprehension. Although the recall protocol is an essay-like instrument, the total score derived is based entirely on summing the discrete propositions correctly recalled. Consequently, item and reliability analyses comparable to those run on multiple-choice tests can and should be performed on recall protocols and on other integrative measures. The purpose of this study is to demonstrate how modified scoring, item and internal consistency analyses, along with Sato's caution indices, can be used to evaluate the quality of the recall protocol as a reading comprehen sion measure. Issues concerning the assumptions underlying classical local independence are discussed with regard to the reading recall protocol and other integrative measures. Results indicate that the procedures can be applied and can yield interpretable results. These results need to be replicated using other texts and other weighting systems. Only when recall protocols are routinely subjected to item and reliability analyses comparable to those performed on other measures, can the instrument be considered a viable alternative.

On the use of the immediate recall task as a measure of second language reading comprehension

“…The immediate written recall task, a method of assessment commonly practised by first language (L1) literacy and psycholinguistic researchers, has been increasingly used in second language (L2) reading research as a measure of comprehension (Lee, 1986;Bernhardt, 1991;Deville and Chalhoub-Deville, 1993;Kobayashi, 1995). It requires readers to read a passage silently and then to write down everything they can remember from the text (Bernhardt, 1983).…”

Investigating the impact of segmented and whole-text repetition on listening comprehension, comprehension processes, and comprehension problems

“…These estimates indicate that the responses to the items are consistent and provide additional information about the quality of the recall protocol measure. In addition, they support Deville and Chalhoub- Deville's (1993) finding that "there is essentially no difference in subjects' relative total scores whether the recalls are scored dichotomously or are weighted" (p. 126).…”

“…Before looking at descriptive and inferential statistics based on these scores, internal consistency estimates are needed "in order to determine whether the total scores are meaningful and appropriate" (Deville and Chalhoub-Deville, 1993, p. 125). However, Deville and Chalhoub-Deville (1993) noted that calculating internal consistency with weighted scores may result in an artificial score-spread and, therefore, an artificially high level of internal consistency. Therefore, the Cronbach's alpha reliability estimate was obtained for the recall protocol using recalls that had been scored both dichotomously and using the weighted rubric.…”

Reading Proficiency Assessment and the ILR/ACTFL Text Typology: A Reevaluation

“…Similar dichotomous scoring methods have been used effectively for assessing reading ability for both immediate recall protocols (Deville & Chalhoub-Deville, 1993) and translations (Buck, 1992).…”