New techniques for usability evaluation of mobile systems

Abstract

Usability evaluation of systems for mobile computers and devices is an emerging area of research. This paper presents and evaluates six techniques for evaluating the usability of mobile computer systems in laboratory settings. The purpose of these techniques is to facilitate systematic data collection in a controlled environment and support the identification of usability problems that are experienced in mobile use. The proposed techniques involve various aspects of physical motion combined with either needs for navigation in physical space or division of attention. The six techniques are evaluated through two usability experiments where walking in a pedestrian street was used as a reference. Each of the proposed techniques had some similarities to testing in the pedestrian street, but none of them turned out to be completely comparable to that form of field-evaluation. Seating the test subjects at a table supported identification of significantly more usability problems than any of the other proposed techniques. However a large number of the additional problems identified using this technique were categorized as cosmetic. When increasing the amount of physical activity, the test subjects also experienced a significantly increased subjective workload. r

A Systematic Evaluation of Mobile Applications for Instant Messaging on iOS Devices

“…The downside of laboratory evaluations is its lack of context (in terms of real usage and environmental influences) [53,54], only present in nearly 11% of the studies [52].…”

The influence of user expertise and prototype fidelity in usability tests

“…For example, if the operation of a car stereo is tested in the form of a single task, the task scenario is characterised by lower breadth than when the operation of a car stereo is part of a multiple-task environment including car navigation. A study comparing mobile phone operation using task scenarios of different breadth revealed that under the single task condition (phone operation while seated at a table), test participants reported more usability problems and lower overall workload than in a dual task condition (phone operation while walking in a pedestrian zone) whereas no difference was found for performance measures (Kjeldskov and Stage, 2004). The depth of a task scenario refers to the level of detail with which a particular task is completed.…”

A Comparison of Pseudo-paper and Paper Prototyping Methods for Mobile Evaluations

“…Mobile systems are typically used by people who are mobile, in dynamically changing, contextually-rich and complex environments; many of the usability problems within mobile application designs are, therefore, best discovered through evaluation of the system in environments representative of the real world [e.g., 4,5,6]. Although mobile system design could potentially benefit from early-stage usability studies based on paper prototypes, such studies are rarely performed due to challenges presented by the mobile use of such prototypes [3].…”

“…This study draws on research in the field of effective mobile, lab-based usability evaluation [e.g., 6,8,9] to expand on the findings of Sá and Carriço [3] and Hendry et al [7] in order to further our understanding of potential mechanisms by which to effectively (and conveniently) use paper prototypes in labbased, mobile evaluations, as well as to discover the relative merits of doing so (as compared to simply employing a traditional seated protocol). The following sections describe our evaluation design and process, and discuss our results, respectively.…”