Criteria For Evaluating Usability Evaluation Methods

Abstract

The current variety of alternative approaches to usability evaluation methods (UEMs) designed to assess and improve usability in software systems is offset by a general lack of understanding of the capabilities and limitations of each. Practitioners need to know which methods are more effective and in what ways and for what purposes. However, UEMs cannot be evaluated and compared reliably because of the lack of standard criteria for comparison. In this article, we present a practical discussion of factors, comparison criteria, and UEM performance measures useful in studies comparing UEMs. In demonstrating the importance of developing appropriate UEM evaluation criteria, we offer operational definitions and possible measures of UEM performance. We highlight specific challenges that researchers and practitioners face in comparing UEMs and provide a point of departure for further discussion and refinement of the principles and techniques used to approach UEM evaluation and comparison.

How can constraint-induced movement therapy for stroke patients be incorporated into the design of a tangible interface? The case study of the ‘Biggest Hit’

“…A formative usability evaluation (Hartson, Andre, & Williges, 2001) during the design process helps to improve the design outcomes. This expert-based evaluation helps uncover usability problems the same way the user would (Hartson et al, 2001). …”

Web accessibility guideline aggregation for older users and its validation

“…These methods differ in terms of their validity, reliability, efficiency, and usefulness; but so far little research has been carried out to analyze these properties. Several studies of usability evaluation methods have shown that user testing methods may fail in yielding consistent results when performed by different evaluators [30,39], and that inspection-based methods are not free of shortcomings either [15,26,49,55]. Although accessibility and usability are two different properties, there is no reason to assume that the kind of uncertainty and mishaps that apply to usability evaluation methods should not apply to accessibility evaluation methods as well.…”

“…This is why methods can be compared in terms of criteria such as validity (the extent to which all and only the true problems are discovered), reliability (the extent to which independent evaluations produce the same results), efficiency (the amount of resources expended to carry out an evaluation that leads to specified levels of effectiveness and usefulness), usefulness (the effectiveness and usability of the produced results) and the method's usability (how easily it can be understood, learned and remembered by evaluators); for more details, the reader is referred to [24,26,49,58]. This paper also shows how validity and reliability, the criteria that characterize a method's effectiveness, change due to aggregation.…”

Characterising user capabilities to support inclusive design evaluation

“…In order to provide useful information and tools to enable design evaluation, it is useful to classify usability evaluation methods into analytical methods and empirical methods [19]. Analytical methods are based on the inspection and analysis of product features, while empirical methods measure user and design performance in actual usage scenarios.…”