Measuring reading behavior in policy documents: a comparison of two instruments

Abstract

Techniques for observing selection and reading behavior in professional documents, such as the thinking-aloud and the click-and-read methods, may affect the reading process to be observed. Such so-called reactivity problems complicate the use of these instruments in experimental research and usability testing. If their influence is unknown, any experimental results obtained with these instruments may be caused by the testing method. One way to detect reactivity effects is to compare different instruments in a series of experimental studies. In this initial study, we compared the thinking-aloud method, the click-and-read method, a combination of these two methods, and a silent reading condition. Subjects read and judged a 53-page policy document in one of these conditions. We investigated whether or not different observation instruments caused specific differences in information selection, judgment, and knowledge. Thinking aloud did not cause any differences in the selection of information. Both the thinking-aloud and the click-and-read methods affected the judgment task outcome. Thinking aloud led to many positive and few negative judgments, whereas silent reading led to many negative and few positive judgments. The results for the click-and-read method showed a tendency toward the same effect. Neither method affected the knowledge test results.

Retrospective think-aloud method

“…This brings this research in line with other studies in which no differences between think-aloud variants were found [26,28,29,30].…”

“…In these studies only minor differences were found between the three variants in terms of output, which makes them largely interchangeable. Ummelen and Neutelings [26] compared concurrent thinking-aloud with the click-and-read method and a silent condition, focusing on reactivity. The different conditions had no effect on information selection and knowledge.…”

Thinking About Thinking Aloud: A Comparison of TwoVerbal Protocols for Usability Testing

“…Specifically, researchers have been investigating to which extent the thinking aloud method may affect the processes being investigated (e.g. [8] [9] [10] [11]). The results are not yet conclusive.…”

Evaluating municipal websites: A methodological comparison of three think-aloud variants

“…Finally, CTA participants might well experience more or less reactivity depending on whether they perform navigational or reading tasks. While reading, the fact that these participants have to think aloud might make them more attentive to the reading material at hand (see also Ummelen and Neutelings (2000) for some evidence to this claim), causing their performance to improve (positive reactivity). While navigating, on the other hand, participants have to think aloud and engage in physical activities, which could increase their cognitive workload and might result in a worse performance (see also Van den Haak et al, 2007).…”