Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison

Abstract

This paper describes a comparative study of three usability test approaches: concurrent thinkaloud protocols, retrospective think-aloud protocols, and constructive interaction. These three methods were compared by means of an evaluation of an online library catalogue, which involved four points of comparison: number and type of usability problems detected; relevance of the problems detected; overall task performance; and participant experiences. The results of the study showed that there were only few significant differences between the usability test approaches, mainly with respect to manner of problem detecting, task performance and participant experience. For the most part, the usability methods proved very much comparable, revealing similar numbers and types of problems that were equally relevant. Taking some practical aspects into account, a case can be made for preferring the concurrent think-aloud protocols over the other two methods. q

Comparing Concurrent and Retrospective Verbal Protocols for Blind and Sighted Users

“…Comparing the two protocols in terms of whether they identify the same problems, we found that only 27 % of the distinct problems were identified by both protocols. Van den Haak et al [25][26][27][28]] also compared overlap between the two protocols in their studies. The overlap in most of the studies [25][26][27] was similar with the overlap reported in the study here, except for one study [28].…”

“…Van den Haak et al [25][26][27][28]] also compared overlap between the two protocols in their studies. The overlap in most of the studies [25][26][27] was similar with the overlap reported in the study here, except for one study [28]. Unfortunately, van den Haak et al did not specifically report the overlap between CVP and RVP, as they included other protocols in their studies.…”

“…A number of studies found that there is no difference between the two protocols in terms of task performance [17,25,26]. However, there are also several studies that showed that verbalization could have an impact on participants performance: either improved [30] or worsened it [29].…”

“…A number of studies compared the number of problems revealed between the two protocols and most of them have demonstrated that the two protocols revealed a comparable set of problems [25][26][27][28]. However, these studies have some limitations, as only one website was used in each one of the studies.…”

“…The thinking aloud may be performed concurrently with conducting the task, known as a concurrent verbal protocol (CVP), or retrospectively while reviewing recordings of their performance on a task, known as a retrospective verbal protocol (RVP). A number of studies have compared these two types of verbal protocol with sighted participants, in terms of the information gathered [3,17] and the number of problems revealed [25,26]. However, a comparison of these protocols when used with blind participants has not yet been performed, in spite of the fact that there are a number of studies which have used verbal protocols with blind participants [8,12,19,20,22].…”