The Evaluator Effect in Usability Studies: Problem Detection and Severity Judgments

Abstract

Usability studies are commonly used in industry and applied in research as a yardstick for other usability evaluation methods. Though usability studies have been studied extensively, one potential threat to their reliability has been left virtually untouched: the evaluator effect. In this study, four evaluators individually analyzed four videotaped usability test sessions. Only 20% of the 93 detected problems were detected by all evaluators, and 46% were detected by only a single evaluator. From the total set of 93 problems the evaluators individually selected the ten problems they considered most severe. None of the selected severe problems appeared on all four evaluators' top-10 lists, and 4 of the 11 problems that were considered severe by more than one evaluator were only detected by one or two evaluators. Thus, both detection of usability problems and selection of the most severe problems are subject to considerable individual variability.

The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods

“…Specifically, the evaluator effect is neither restricted to novice evaluators nor to evaluators knowledgeable of usability in general. The evaluator effect was also found for evaluators with experience in the specific UEM they have been using (Jacobsen et al, 1998;Lewis et al, 1990;Molich et al, 1998Molich et al, , 1999. Furthermore, the evaluator effect is not affected much by restricting the set of problems to only the severe problems.…”

“…Hence, the evaluators were requested to de-tect problems according to the nine criteria, and they were asked to report time-stamped evidence and a free-form description for each problem. Based on the evaluators' problem lists, two of the authors of Jacobsen et al (1998) independently constructed a master list of unique problem tokens. They agreed on 86% of the problem tokens, and by discussing their disagreements and the problems they did not share, a consensus was reached.…”

The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods

“…Previous studies (e.g., Hertzum & Jacobsen, 1999;Jacobsen, Hertzum, & John, 1998;Nielsen, 1992) have used the average detection rate of a single evaluator as their basic measure of the evaluator effect. This measure relates the evaluators' individual performances to their collective performance by dividing the average number of problems detected by a single evaluator by the number of problems detected collectively by all the evaluators.…”

Evidence Based Design of Heuristics for Computer Assisted Assessment

“…Evaluators may have difficulty in distinguishing between a problem that is rated as a Major Usability Problem and a Usability Catastrophe when using playability heuristics. When comparing evaluator's classification of problems to severity ratings research has shown that inter-rater reliability tends to be low [19]. This may be due to the difficulty evaluators have in distinguishing the boundaries between scales.…”