Changing Analysts’ Tunes: The Surprising Impact of a New Instrument for Usability Inspection Method Assessment

Cockton, Gilbert; Woolrych, Alan; Hall, Lynne; Hindmarch, Mark

doi:10.1007/978-1-4471-3754-2_9

Cited by 29 publications

(44 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Relating to the first issue, i.e. guidance of heuristics in discovering a problem, in a study where an extended problem report format was used, Cockton et al (2003b) found significant improvements in appropriateness scores increasing to a mean score of 61% from an earlier one of 31% . They contributed at least part of the difference to discovery methods and part to the extended problem report format.…”

Section: Results According To Heuristicsmentioning

confidence: 99%

“…In heuristic evaluation, the portion of the application covered is up to the evaluator, especially if he/she is given total freedom of what aspects to cover. In a study of discovery methods, Cockton et al (2003b) report that most evaluators choose to use system-searching or system-scanning more often than goalplaying and method-following. In a think-aloud user-testing, the coverage is influenced by the set of tasks presented to the user.…”

Section: Task Selectionmentioning

confidence: 99%

“…Comparing two independent usability problem sets from two UEMs has drawbacks. An evaluator may predict problems that are miscoded as False Alarms since a user never had an opportunity to see them during user testing because he or she did not visit that part of the application Cockton et al, 2003bCockton et al, , 2004Woolrych et al, 2004). Cockton and Woolrych (2001) suggest to systematically derive task sets for user testing from the initial set of predicted problems identified in heuristic evaluation so as to increase the power of user testing for exposing all predicted problems that really exist, i.e.…”

Section: Task Selectionmentioning

confidence: 99%

See 2 more Smart Citations

Heuristic evaluation: Comparing ways of finding and reporting usability problems

Hvannberg

Law

Lárusdóttir

2007

Interacting with Computers

140

View full text Add to dashboard Cite

Research on heuristic evaluation in recent years has focused on improving its effectiveness and efficiency with respect to user testing. The aim of this paper is to refine a research agenda for comparing and contrasting evaluation methods. To reach this goal, a framework is presented to evaluate the effectiveness of different types of support for structured usability problem reporting. This paper reports on an empirical study of this framework that compares two sets of heuristics, Nielsen's heuristics and the cognitive principles of Gerhardt-Powals, and two media of reporting a usability problem, i.e. either using a web tool or paper. The study found that there were no significant differences between any of the four groups in effectiveness, efficiency and inter-evaluator reliability. A more significant contribution of this research is that the framework used for the experiments proved successful and should be reusable by other researchers because of its thorough structure.

show abstract

Section: Results According To Heuristicsmentioning

confidence: 99%

Section: Task Selectionmentioning

confidence: 99%

Section: Task Selectionmentioning

confidence: 99%

See 1 more Smart Citation

Heuristic evaluation: Comparing ways of finding and reporting usability problems

Hvannberg

Law

Lárusdóttir

2007

Interacting with Computers

140

View full text Add to dashboard Cite

show abstract

“…Building upon the work of Lavery et al, Cockton et al (2003) showed how an extended reporting format made evaluators using heuristic evaluation predict fewer false positives, compared to a previous study using a simpler reporting format. The extended reporting format asked evaluators to report problem descriptions, the likely/actual difficulties, the context of a problem, and assumed causes.…”

Section: Related Workmentioning

confidence: 96%

“…The aim of this documentation of problems was to allow a multi-facetted description of problems without going into the detail of the problem reporting formats of Lavery et al (1997) or Cockton et al (2003). …”

Section: Problem Identificationmentioning

confidence: 99%

Comparison of techniques for matching of usability problem descriptions

Hornbæk

Frøkjær

2008

Interacting with Computers

View full text Add to dashboard Cite

a b s t r a c tMatching of usability problem descriptions consists of determining which problem descriptions are similar and which are not. In most comparisons of evaluation methods matching helps determine the overlap among methods and among evaluators. However, matching has received scant attention in usability research and may be fundamentally unreliable. We compare how 52 novice evaluators match the same set of problem descriptions from three think aloud studies. For matching the problem descriptions the evaluators use either (a) the similarity of solutions to the problems, (b) a prioritization effort for the owner of the application tested, (c) a model proposed by Lavery and colleagues [Lavery, D., Cockton, G., Atkinson, M.P., 1997. Comparison of evaluation methods using structured usability problem reports. Behaviour and Information Technology, 16 (4/5), 246-266], or (d) the User Action Framework [Andre, T.S., Hartson, H.R., Belz, S.M., McCreary, F.A., 2001. The user action framework: a reliable foundation for usability engineering support tools. International Journal of Human-Computer Studies, 54 (1), 107-136].The resulting matches are different, both with respect to the number of problems grouped or identified as unique, and with respect to the content of the problem descriptions that were matched. Evaluators report different concerns and foci of attention when using the techniques. We illustrate how these differences among techniques might adversely influence the reliability of findings in usability research, and discuss some remedies.

show abstract

A Structured Expert Evaluation Method for the Evaluation of Children’s Computer Games

Baauw

Bekker

Barendregt

2005

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Inspection-based evaluation methods predicting usability problems can be applied for evaluating products without involving users. A new method (named SEEM), inspired by Norman's theory-of-action model [18] and Malone's concepts of fun [15], is described for predicting usability and fun problems in children's computer games. This paper describes a study to assess SEEM's quality. The results show that the experts in the study predicted about 76% of the problems found in a user test. The validity of SEEM is quite promising. Furthermore, the participating experts were able to apply the inspection-questions in an appropriate manner. Based on this first study ideas for improving the method are presented.

show abstract

Changing Analysts’ Tunes: The Surprising Impact of a New Instrument for Usability Inspection Method Assessment

Cited by 29 publications

References 5 publications

Heuristic evaluation: Comparing ways of finding and reporting usability problems

Heuristic evaluation: Comparing ways of finding and reporting usability problems

Comparison of techniques for matching of usability problem descriptions

A Structured Expert Evaluation Method for the Evaluation of Children’s Computer Games

Contact Info

Product

Resources

About