Assessing Text Processing: A Comparison of Four Methods

Abstract

Tests commonly used to determine progress in reading skill and proficiency typically assess reading products (e.g., identifying characters or factual information, sequencing events) rather than the reading processes used to generate responses (e.g., hypothesizing, evaluating, monitoring, questioning). Yet, effective processing often determines how successfully a reader responds on testing measures. Identifying measures that can assist educators to better understand how a student processes text is vital. The purpose of this research was to compare the data generated from 4 assessment methods used to evaluate how readers process text: think-aloud, interview, error detection, and questionnaire. In this descriptive study, 40 fourth-grade students of average reading ability read the same text, and data were collected as each student responded to 1 of the 4 assessment measures. Results indicated that students assessed with think-aloud and interview measures generated a greater number and broader range of text processing responses. Thinkaloud protocols reflected a close interaction with text while interview responses included more evidence of metacognitive processing. Issues frequently identified as problematic in error detection research (e.g., difficulty finding errors, purpose for reading) are supported in this study. The questionnaires provide less specific data about individual text processing since students were limited by the answer choices for each question. Results from this study suggest that by using think-aloud and interview assessments, educators can obtain a more complete understanding of a reader's text processing skill than by using error-detection and questionnaire methods.

A Think-Aloud Protocols as a Cognitive Strategy to Increase Students’ Writing Narrative Skill at Efl Classroom

“…However, Beck and Kucan (1997) and thoughts (Scott, 2008). In this way, text processing and learning activities are directly revealed without delay and are expressed in students' own words.…”

“…Significant research has focused on explicit efforts to understand the thinking process and the comprehension of text (Davey, 1983; Bereiter and Bird, 1985). Utilizing think-alouds in such a manner involves teacher modeling, teacher-student interaction, and finally, the independent use by the student.However, Beck and Kucan (1997) and thoughts (Scott, 2008). In this way, text processing and learning activities are directly revealed without delay and are expressed in students' own words.…”

“…Although think-alouds provide scaffolding for students as they engage in higher order thinking (Oster, 2001), a full assessment of their thinking process is limited to what is openly shared in the verbal exchange. Many attempts have been made in the literature to measure learning strategies in various contexts with different data gathering methods (Schellings, 2011;Scott, 2008).…”

User-centered design for personalization

“…In order to reveal the full spectrum of a system's strong and weak points, one needs to evaluate it using multiple methods (Ebling & John, 2000;Peleg, Shackak, Wang, & Karnieli, 2009;Scott, 2008;Zabed Ahmed, 2008). However, the sets of issues the different methods elicit may overlap and, as a result, the added value of applying an extra method may be limited.…”

“…Other comparisons have addressed the differences between thinking-aloud, questionnaires and/or interviews when applied to tasks or systems without personalized features. In the case of text-processing, Scott (2008) found that thinking-aloud and interviews elicit the same responses. Meanwhile, other researchers did find differences between the methods.…”