What Are You Thinking? Postsecondary Student Think-Alouds of Scientific and Quantitative Reasoning Items

Cognitive Processing of Self-Report Items in Educational Research: Do They Think What We Mean?

“…There may be elements in common with other sources of evidence for validity, yet the proposed category seems quite distinct. As discussed by Thelk and Hoole (2006), cognitive validity relates directly to Messick's substantive validity, which depends on the congruence between the assumed and actual cognitive processes that test takers experience, an approach also suggested by Pellegrino, Chudowsky, and Glaser (2001). Those applications of cognitive validity, however, are more germane to the assessment of knowledge and skills-analogous to Piaget assessing the child-whereas the suggested use in our context focuses on assessing characteristics of a survey item.…”

“…This view is changing, however, with the increasing application of rigorous methods to gather and analyze data directly from members of targeted respondent populations. The term cognitive validity has been used to designate such evidence in the domain of performance assessment in math and science (Ruiz-Prima, Schultz, Ki, & Shavelson, 2001;Thelk & Hoole, 2006). In that domain the issue is whether respondents' cognitive processes during test performance mirror those intended by test designers, which in Messick's approach would be classified as evidence related to substantive validity (Yu & Wolters, 2002).…”

Using Metacognitive Strategies to Help Students Learn in Pretest and Posttest Review

“…27 There have been instances when the wording of a question did not measure what we thought it would. ''I have a better understanding of where the students went wrong in their thinking.''…”

“…27 They used the think-out-loud strategy with students completing multiple-choice test items to help them establish validity for those items. 27 They used the think-out-loud strategy with students completing multiple-choice test items to help them establish validity for those items.…”

Program Evaluation

“…15 Research supports that thinking aloud facilitates student learning and, in addition, helps the teacher decide whether the test items are being analyzed in the intended manner. 16 As reported by one student, ''I figured out my best learning style by talking it [test item] throughII need interactive, participatory activities. I have to go through a thinking process.''…”