Cueing Retrospective Verbal Reports in Usability Testing Through Eye-Movement Replay

Abstract

An experimental validation is presented of a novel method for usability testing that entails the playback of dynamic eyetracking data to cue the elicitation of retrospective verbal reports. Participants in our study produced: (1) think-aloud reports during an online search task, and (2) retrospective reports during another online search task, with reports being cued by the playback of either the screen capture of events or the participant's own eye-movements. Task-completion times and response rates were recorded for all reporting methods. Fewer participants completed the search task whilst thinking aloud, indicating the reactivity of this technique. Verbal transcripts were coded for instances of usability problems. The eye-cued method identified more usability problems than the think-aloud or screen-cued methods. A significant interaction between search engine type and retrospective cue type suggests that the value of the eye-cue method for eliciting usability problems may be greatest with more complex search environments. Our results demonstrate that when cued appropriately, retrospective reports may be less reactive and more informative than other verbalisation techniques.

Comparing different eye tracking cues when using theretrospective think aloud method in usability testing

“…• Think aloud processes may not be sufficient since certain cognitive processes are unconscious and participants may not be able to adequately verbalize their thought process [5].…”

“…• For CTA methods, an issue is that cognitive processes are quicker than verbal processes, so participants might be thinking about more than they are able to verbally express [5].…”

“…Using a video cue that features eye movements (a gaze video replay) has been demonstrated as more effective at eliciting comments from users than an uncued RTA [17]. Showing a playback of participants' eye-movements overlaid on a video showing the steps they took while completing a task has proven to be a successful way to elicit information from the participants and, in addition, allows for an accurate measure of other variables, such as task time [5] [21].…”

“…The goal of this study is to compare four different retrospective think aloud methods in a website usability test: RTA without any cue, RTA with a video cue (screen video), RTA with gaze plot cues (superimposed eye movements on still images), and RTA with a gaze video cue (superimposed eye movements on a screen video). Previous research has, been done on using no cue, video replay and gaze video replay, but not on using a static gaze image as a cue when doing RTA [5] [10]. The paper is structured as follows: Initially, a brief introduction to eye tracking is given followed by a short presentation of the think aloud methodology as well as why RTA is seen as a suitable method in combination with eye tracking.…”

An Analysis of Students’ Cognitive Bias in Experimental Activities Following a Lab Manual

“…RVP collection can be used as a complement to the drawbacks of CVP collection. In fact, the cognitive process is faster than the verbal process, and hence, the research participants may actually be thinking more than during the verbal process, and because certain cognitive processes are unconscious, they may not be adequately expressed or may be totally ignored [34]. RVP collection is needed to offset these shortcomings and to complement the cognitive process represented by CVPs [35].…”