Abstract. Evaluating the usability of mobile systems raises new concerns and questions, challenging methods for both lab and field evaluations. A recent literature study showed that most mobile HCI research projects apply lab-based evaluations. Nevertheless, several researchers argue in favour of field evaluations as mobile systems are highly context-dependent. However, field-based usability studies are difficult to conduct, time consuming and the added value is unknown. Contributing to this discussion, this paper compares the results produced by a laboratory-and a field-based evaluation of the same context-aware mobile system on their ability to identify usability problems. Six test subjects used the mobile system in a laboratory while another six used the system in the field. The results show that the added value of conducting usability evaluations in the field is very little and that recreating central aspects of the use context in a laboratory setting enables the identification of the same usability problem list.

No abstract

No abstract

Abstract. It is often argued that usability problems should be identified as early as possible during software development, but many usability evaluation methods do not fit well in early development activities. We propose a method for usability evaluation of use cases, a widely used representation of design ideas produced early in software development processes. The method proceeds by systematic inspection of use cases with reference to a set of guidelines for usable design. To validate the method, four evaluators inspected a set of use cases for a health care application. The usability problems predicted by the evaluators were compared to the result of a conventional think-aloud test. About one fourth of the problems were identified by both think-aloud testing and use case inspection; about half of the predicted problems not found by think-aloud testing were assessed as providing useful input to early development. Qualitative data on the evaluators' experience using the method are also presented. On this background, we argue that use case inspection has a promising potential and discuss its limitations.

Evaluating mobile technologies "in the real world" is hard. It is challenging to capture key situations of use, hard to apply established techniques such as observation and "thinking aloud", and it is complicated to collect data of an acceptable quality. In response to these challenges, we have developed a "field laboratory" for evaluating mobile technologies in situ. Facilitating high-quality data collection as well as unobstructed user interaction, the field laboratory allows a small wireless camera to be attached to a mobile device, capturing a close-up image of the screen and buttons. This chapter describes the iterative development of our field laboratory over 4 years of evaluating several mobile systems in field settings. It leads to a description of the current setup and how it is used, and explains the rationales for key decisions on technology and form factors made throughout its development. INTRODUCTIONStudying peoples' use of technology is a key activity within the research field of HumanComputer Interaction (HCI) providing software developers with invaluable information about the usability and usefulness of their systems at different stages of the process from conceptual design to a final implementation. Traditionally, such studies have taken place in dedicated "usability laboratories" where users' interaction with computer systems can be observed in a controlled experimental setting providing video and audio data of very high quality. Studying the usability of mobile technologies, however, raises new questions and concerns. Mobile systems are typically used in highly dynamic contexts involving a close interaction between people, systems and their surroundings. Therefore, studying mobile technology use in situ seems like an appealing or even indispensable approach -rather than trying to recreate the use situation realistically in a laboratory. However, studying mobile technology usability "in the real world" is difficult. It is difficult to capture key situations of use, apply established usability techniques such as observation and "thinking aloud" without interfering with the situation, and it is complicated to collect data of an acceptable quality.In response to some of these challenges, we have extended our stationary usability laboratory at Aalborg University's Department of Computer Science with a mobile counterpart, the field laboratory, which can be taken into the field when studying mobile system use and usability. Facilitating high-quality data collection as well as unobstructed user interaction, the field laboratory allows a small wireless camera to be attached to the mobile device, capturing a close-up image of the screen and buttons while a third-person view is captured by a handheld camcorder.The purpose of this chapter is to communicate our experiences with developing and using the field laboratory for evaluating mobile technology use and usability in situ by taking the readers through four years of major iterations leading to its current configuration. By doing this, it is ou...