Developing an usability test to evaluate the use of augmented reality to improve the first interaction with a product

Abstract

Developing an usability test involves defining the metrics that are being evaluated, the methods used and the right tasks to be performed to achieve the objectives from the usability test. This article presents the development of an usability test aimed to evaluate the experience while first interacting with a product by using the augmented reality technology. This usability test is part of a bigger research that aims to evaluate if the use of augmented reality helps the process of learning how to use a new product. The paper starts with a literature review of related works, explaining many of the tools developed for evaluating usability tests. This section is followed by a methodological explain and the description of the developed usability test itself. The conclusion presents suggestions of further improvements.

A usability test protocol for evaluating mixed reality environments

“…One of the most widely used usability questionnaires is the USE (Usefulness, Satisfaction, and Ease of Use) Questionnaire (Lund, 2001). It is developed to test the usability of software, hardware, services, user support materials, and products (i.e., portable vacuum cleaner (Albertazzi, Okimoto and Ferreira, 2012) under four categories: Usefulness, Satisfaction, Ease of Use, and Ease of Learning. It includes generic statements that evaluate the product, such as, "it helps me be more productive, "it is easy to learn to use it", and "I would recommend it to a friend".…”

Evaluation of User Experience in Augmented Reality Mobile Applications

“…Analyzing tools for augmented reality user experience evaluation, instruments such as SUS (System Usability Scale), USE (Usefulness, Satisfaction, Easy to Use and to Learn), HARUS (Handheld Augmented Reality Usability Scale), usability questionnaire by ISO (Usability, Effectiveness, Efficiency), TAM (Technology Acceptance Model), UEQ (User Experience Questionnaire) are usually applied in scientific research (Albertazzi et al, 2012;Davidavičienė et al, 2019;Guimaraes & Martins, 2014;Hinderks et al, 2019;Mota et al, 2018;Pantano et al, 2017;Rese et al, 2017;Santoso et al, 2016;Schrepp et al, 2014Schrepp et al, , 2017aSchrepp et al, , 2017bWang et al, 2015) (Table 1). After a detailed examination of the augmented reality user experience tools and their advantages and disadvantages (Table 1), the User experience questionnaire evaluation tool was selected due to the complex questionnaire design, six evaluated dimensions: attractiveness (overall user impression of the object being evaluated); efficiency (is it possible to use the product quickly and efficiently, does the user interface look organized); perspicuity (is it easy to understand how to use the product); dependability (whether the user feels in control of the interaction or whether the interaction with the product is safe and predictable); stimulation (whether it is interesting and exciting to use the product or whether the consumer feels motivated to continue using the product); novelty (whether the product design is innovative and creative or whether the product attracts the consumer's attention).…”

User Experience Evaluation and Creativity Stimulation With Augmented Reality Mobile Applications

“…In evaluating UE, researchers use the following key evaluation tools as system usability scale (SUS), usefulness, satisfaction, and ease of use, usage questionnaire according to International Organization for Standardization (ISO) (usability, effectiveness, efficiency), technology acceptance model (TAM), user experience questionnaire (UEQ). SUS is a system of statement-based surveys with a 5-point scale, when the user, after the use of the system, is asked 10 questions analyzing the main processes of product useease of use, fluidity, comprehensibility, satisfaction (Albertazzi, Okimoto, & Gitirana Gomes Ferreira, 2012).…”

“…Usefulness, satisfaction, and ease of use (USE) questionnaire is based on criteria with 7 levels and consists of 30 questions that analyze the product in key dimensions such as usefulness, ease of use, ease of learning, and satisfaction. Two open questions are also asked where the user is requested to identify the main negative and positive aspects of the subject being evaluated (Albertazzi et al, 2012).…”