Evaluating Context-Aware Mobile Interfaces for Professionals

Abstract

Evaluation refines and validates design solutions in order to establish adequate user experiences. For mobile user interfaces in dynamic and critical environments, user experiences can vary enormously, setting high requirements for evaluation. This chapter presents a framework for the selection, combination, and tuning of evaluation methods. It identifies seven evaluation constraints, that is, the development stage, the complexity of the design, the purpose, participants, setting, duration, and cost of evaluation, which influence the appropriateness of the method. Using a combination of methods in different settings (such as Wizard-of-Oz, game-based, and field evaluations) a concise, complete, and coherent set of user experience data can be gathered, such as performance, situation awareness, trust, and acceptance. Applying this framework to a case study on context-aware mobile interfaces for the police resulted in specific guidelines for selecting evaluation methods and succeeded to capture the mobile context and its relation to the user experience.

Effects of mobile map orientation and tactile feedback on navigation speed and situation awareness

“…We distinguish four research questions: Recently, game-based simulation has become popular in the scientific community, because of the possibilities and control it provides [11,8,19]. Logging data for analysis is one of the other advantages of experimenting in simulated environments.…”

Balancing Awareness and Interruption in Mobile Patrol using Context-Aware Notification

“…Regarding participants, end-users may be employed in all stages of the development process, depending on their availability (Streefkerk et al, 2008b). However, access to police end-users is limited, making it more cost-effective to only employ them at select moments.…”

“…In evaluating context-aware systems, the evaluation setting, participants and metrics should be chosen carefully (Streefkerk, van Esch-Bussemakers, Neerincx & Looije, 2008b). For the two studies in this paper both the fidelity and realism of the evaluation stetting are important (Smets et al, 2010).…”

“…In the current mobile experiment, evaluation metrics are based on criteria from the police domain (Streefkerk et al, 2008b). For example in police patrol, fast responses to high priority incidents are important.…”