Concurrent Think-Aloud Verbalizations and Usability Problems

Abstract

The concurrent think-aloud protocol—in which participants verbalize their thoughts when performing tasks—is a widely employed approach in usability testing. Despite its value, analyzing think-aloud sessions can be onerous because it often entails assessing all of a user's verbalizations. This has motivated previous research on developing categories to segment verbalizations into manageable units of analysis. However, the way in which a category might relate to usability problems is currently unclear. In this research, we sought to address this gap in our understanding. We also studied how speech features might relate to usability problems. Through two studies, this research demonstrates that certain patterns of verbalizations are more telling of usability problems than others and that these patterns are robust to different types of test products (i.e., physical devices and digital systems), access to different types of information (i.e., video and audio modality), and the presence or absence of a visualization of verbalizations. The implication is that the verbalization and speech patterns can potentially reduce the time and effort required for analysis by enabling evaluators to focus more on the important aspects of a user's verbalizations. The patterns could also potentially be used to inform the design of systems to automatically detect when in the recorded think-aloud sessions users experience problems.

Participatory Design of an Electronic Medical Record for Paediatric Palliative Care: A Think-Aloud Study with Nurses and Physicians

“…In CTAs, participants are asked to verbalize their feelings and emotions directly while interacting with a (new) technology [36,37]. Through this, the "black box" of emotional and cognitive processes as well as reactions can be observed and evaluated; recall problems can potentially be minimized [38,39].…”

“…To analyse the learning ability and the self-explanatory capability of the PCM, only a short overview of its central functions and the plus button was explained. During the execution of the tasks, interactions with participants were kept to a minimum [36]. When questions or problems arose during testing, specific contextual responses or queries were made by the lead researcher.…”

Participatory Design of an Electronic Cross-Facility Health Record (ECHR) System for Pediatric Palliative Care: A Think-Aloud Study

“…A brief introduction to the ECHR system and its basic logic was given (File S3). Participants were also informed that they should first try to solve problems on their own and verbalize this accordingly [23]. Some of the system views were not self-explanatory due to their unique logic.…”

“…For this purpose, the proven methodology of concurrent think-aloud (CTA) was used. In CTA, participants are asked to verbalize their thoughts while using software [23,24]. Using CTA made it possible to capture the cognitive and emotional reactions and processes of participants while using software [25].…”

Video conferencing as a face-to-face online meeting app: user preference based on usability testing

“…Several other usability evaluation methods have been implemented, such as the think-aloud protocol and eye-tracking technology for usability research. Despite its value, analyzing think-aloud sessions can be onerous because it often entails assessing all users' verbalizations [28]. Then, eye tracking is a valuable method for analyzing user behavior and uncovering potential problems with a website or app design.…”