The Value of Thinking-Aloud Protocols in Industry: A Case Study at Microsoft Corporation

Abstract

Thinking-aloud protocols traditionally have been used by academic researchers as a qualitative data collection method. This method is currently gaining acceptance in industry usability testing. The Usability Group at Microsoft has adopted the thinking-aloud protocol as a primary method for obtaining data from users. We have found the method valuable not only because it is valid for gathering qualitative data, but also because it is responsive to the constraints we face and the organizational culture we work within. The issue of validity has been discussed in detail by researchers such as Deffner & Rhenius and Ericcson & Simon. Our case study further pursues the validity of thinking-aloud protocols and also discusses how this method allows the researcher to work within industry constraints and incorporate changes into the product within a small time frame. Finally, our case study demonstrates how thinking-aloud protocols fit in well with Microsoft corporate culture where understandable and persuasive results are needed. This case study will have particular relevance for usability practitioners in industry.

Collecting Feedback during Software Engineering Experiments

“…It has also become a valuable research method in several applied disciplines. For example, it has been used in medicine to identify strategies used by experts and to evaluate the effects of new technologies on decision-making processes (Jungk et al, 2000;Patel et al, 2001); in education to identify strategies and processes used in learning environments (Davidson et al, 1996;Garner, 1988;Nathan, 1991); in software engineering for development of a comprehension model (von Mayrhauser and Lang, 1999); and in usability evaluation (Boren and Ramey, 2000;Denning et al, 1990;Haak and Jong, 2003).…”

“…The approach we have used could be considered as a form of experience sampling (Larson and Csikszentmihalyi, 1983) although the constrained experimental context and emphasis on problem solving in our work gives a much more focussed environment for collecting experience data. Thinkaloud methods have proved useful for collecting data concerning subjects in cognitive and usability studies (Anderson, 1987;Denning et al, 1990;Ericsson and Simon, 1993). These methods typically involve human observers and are expensive regarding time spent by observers in addition to that used by the subjects.…”

The Added Usefulness of Process Measures Over Performance Measures in Interface Design

“…Studies have been conducted on the use of think-aloud techniques to be used for collecting process data (Deffner & Yuasa, 1994;Denning, Hoiem, Simpson, & Sullivan, 1990;Page & Rahimi, 1995;Rhenius, 1990;Wright & Monk, 1991), with the findings by Page and Rahimi (1995) supporting the practice of collecting retrospective verbal reports to obtain information about mistakes and complex design problems. Research has shown the effectiveness of both eye movement and retrospective verbal protocol, whereas the comparison of verbalization and eye movement data has shown that eye movement and the cognitive process go together (Rhenius, 1990).…”

How usability testing can aid the development of online documentation

“…Structured Walk-Througs Directed Response --Thiniking aloud protocols (Sullivan, 1987;Denning et al (1990) --Question-asking protocols (Kato, 1989) --Co-discovery Learning (Kennedy, 1989) Involves closely scripted tasks, often guided by test observer.…”