It is widely discussed whether the usefulness of mobile systems and services should be evaluated in the field or in the lab. The aim of this paper is to make a case for conducting longitudinal field trials when evaluating the usefulness of mobile services, and to partly base such evaluations on quantitative usage data automatically captured on the device. A framework for automatic capture and analysis of such data is introduced, and it is shown how it has been used to evaluate a mobile service for diabetes management. Results and experiences based on a three month field trial with a single diabetic user is presented and it is demonstrated how these results can be used to make inferences about the use of the service with regard to functionality and usage patterns.

The gap between how the academic world develops usability and user experience (UX) methods, and how the industry employs these methods is perceived as both broad and deep. But is that the real picture -and has there been a change in how companies work within these fields over the past two years? By conducting interviews with eight companies, this paper tries to answer these two questions. The companies were initially interviewed in 2013 and by follow-up interviews in 2015 the paper draws a picture of how the companies work with UX and usability in an agile development environment. We identify the challenges they are facing and if, and how the work progresses. We found that the UX maturity during these two years had changed significantly. This was revealed by the fact that almost all of the companies in 2015 had implemented or were in the process of developing a UX strategy together with more formalized UX processes. They also allocated more resources to conduct UX and usability work than earlier. We found that all of the companies made use of low-fi prototyping, followed by usability testing, workshops, personas, expert evaluations, user or customer journeys, customer visits and user task analyses. Almost all the companies carried out development using the Scrum framework. All of the companies were interested in the idea of agile UX, and found the idea of using the developers as a UX resource interesting. This, together with an idea of modifying existing usability methods to be used in an agile, industrial setting could be a solution to bridge the gap between academia and the industry.

This paper presents results and conclusions about the current evaluation methodologies for Spoken Dialogue Systems (SDS). The PARADISE paradigm, used for evaluation in the DARPA Communicator project is briefly introduced and discussed through the application to the OVID home banking dialogue system. It is shown to provide results consistent with those obtained by the DARPA community, but a number of problems and limitations are pointed out.The issue of user attitude measures through questionnaires is discussed. This is an area that have not received much attention from the speech technology community, but is important in order to obtain valid results and conclusions about usability A general presentation of the issues that must be addressed when developing and employing questionnaires is given with a focus on how to ensure the reliability and validity of the results. Examples of results obtained from the OVID project are used to illustrate this.

No abstract

No abstract