Comparison of evaluation methods using structured usability problem reports

Abstract

Recent HCI research has produced analytic evaluation techniques which claim to predict potential usability problems for an interactive system. Validation of these methods has involved matching predicted problems against usability problems found during empirical user testing. This paper shows that the matching of predicted and actual problems requires careful attention, and that current approaches lack rigour or generality. Requirements for more rigorous and general matching procedures are presented. A solution to one key requirement is presented: a new report structure for usability problems. It is designed to improve the quality of matches made between usability problems found during empirical user testing and problems predicted by analytic methods. The use of this report format is placed within its design research context, an ongoing project on domain-speci® c methods for software visualizations.

Combining situated Cognitive Engineering with a novel testing method in a case study comparing two infusion pump interfaces

“…Deviations from the normative path were also used as guidance for the qualitative analysis, as this allowed for efficient identification of potentially critical incidences (those with strong deviations). Specific types of deviations and errors were isolated and described using structured report forms (Lavery et al, 1997) .…”

Criteria For Evaluating Usability Evaluation Methods

“…Researchers who have studied usability problem extraction, description, and classification Lavery, Cockton, & Atkinson, 1997) have made just this kind of separation: regarding these as separate functions of usability engineering support tools for classification and reporting, used in conjunction with UEMs. In fact, most methods for problem description and reporting are independent of the evaluation method and can be used with any UEM.…”

Comparing Concurrent and Retrospective Verbal Protocols for Blind and Sighted Users

“…In the first phase of analysis, the problems identified by the users were structured using a variation of the model of Lavery et al [11], in which the problems are analysed in relation to four components: cause, breakdown, outcome and design change. For this study, the design change component was not used.…”