Users’ design feedback in usability evaluation: a literature review

Abstract

IntroductionInvolving users in usability evaluation is valuable when designing information and communication technology (ICT), and a range of usability evaluation methods (UEM) support user involvement. Relevant methods include adaptations of usability testing [1], usability inspection methods such as pluralistic walkthrough [2], and inquiry methods such as interviews [3], and focus groups [4].Users involved in usability evaluation may generate two types of data. We term these interaction data and design feedback. Interaction data are recordings of the actual use of an interactive system, such as observational data, system logs, and data from think-aloud protocols. Design feedback are data on users' reflections concerning an interactive system, such as comments on experiential issues, considerations of the system's suitability for its context of use, usability problem predictions, and design suggestions. AbstractAs part of usability evaluation, users may be invited to offer their reflections on the system being evaluated. Such reflections may concern the system's suitability for its context of use, usability problem predictions, and design suggestions. We term the data resulting from such reflections users' design feedback. Gathering users' design feedback as part of usability evaluation may be seen as controversial, and the current knowledge on users' design feedback is fragmented. To mitigate this, we have conducted a literature review. The review provides an overview of the benefits and limitations of users' design feedback in usability evaluations. Following an extensive search process, 31 research papers were identified as relevant and analysed. Users' design feedback is gathered for a number of distinct purposes: to support budget approaches to usability testing, to expand on interaction data from usability testing, to provide insight into usability problems in users' everyday context, and to benefit from users' knowledge and creativity. Evaluation findings based on users' design feedback can be qualitatively different from, and hence complement, findings based on other types of evaluation data. Furthermore, findings based on users' design feedback can hold acceptable validity, though the thoroughness of such findings may be questioned. Finally, findings from users' design feedback may have substantial impact in the downstream development process. Four practical implications are highlighted, and three directions for future research are suggested. RESEARCHFølstad Hum. Cent. Comput. Inf. Sci. (2017) Cent. Comput. Inf. Sci. (2017) 7:19 The value of interaction data in evaluation is unchallenged. Interaction data is held to be a key source of insight in the usability of interactive systems and has been the object of thorough scientific research. Numerous empirical studies concern the identification of usability problems on the basis of observable user behaviour [5]. Indeed, empirical UEM assessments are typically done by comparing the set of usability problems identified through the assessed UEM with a ...

Can users recall their user experience with a technology? Temporal bias and the system usability scale.

“…However, this may have effected the dropout rate. Participants reported some usability challenges [14]. This potentially answers why many of the SUS scores were low.…”

“…Koon et al [12] also explores the utilitarian, hedonic and social aspects of smartphones to measure people continually engage in smartphone activity. We wish to further this work and differentiate by looking at the user experience of an application over time using SUS survey [13,14].…”

How People Judge the Usability of a Desktop Graphic User Interface at Different Time Points: Is there Evidence for Memory Decay, Recall Bias or Temporal Bias?

“…Koon et al [16] also explores the utilitarian, hedonic and social aspects of smartphones to measure how users continually engage in smartphone activity. We wish to further this work and differentiate by looking at the user experience of a technology over time using the SuS survey [17,18]. User opinions in the moment of using a technology and retrospectively are likely to be different which is why some researcher prefer to use ecological momentary assessment [19].…”

“…Add a grade for a non-Turnitin student assessment. 18 Add grades for all students on a non-Turnitin student assessment. 19 Use the grading functionality to create a calculation which sums the Turnitin and non-Turnitin assessments.…”

Evaluating Web and Mobile User Interfaces With Semiotics: An Empirical Study

“…In fact, usability refers to the extent to which a software system, or any product, can be used efficiently, effectively and satisfactorily within a specified user context [3]. The evaluation of usability is one of the main cornerstones of designing intuitive user interfaces [4], [5]. Although, personal computers with graphical user interfaces have been widely used since the 1980's, due to technological innovations, societal changes and dynamic user preferences, our knowledge regarding UI design in the domain of human-computer interaction (HCI) is in need of continuous improvement [6]- [8].…”