Usability Evaluation of Artificial Intelligence-Based Voice Assistants: The Case of Amazon Alexa

Zwakman, Dilawar Shah; Pal, Debajyoti; Arpnikanondt, Chonlameth

doi:10.1007/s42979-020-00424-4

Cited by 50 publications

(37 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The evaluation of home assistants frequently favors more technical tests related to speech identification and/or interaction response times [ 34 , 36 , 38 , 43 ]. An exception is the evaluation performed by Rahman and coworkers [ 40 ], which uses the System Usability Scale (SUS), a highly graphical user interface (GUI) oriented evaluation tool [ 67 ]. Kocaballi and coworkers compared different questionnaires for evaluating the user experience in conversational interfaces, including SUS and questionnaires specific for voice interfaces, and recommend using multiple questionnaires for a more complete evaluation [ 68 ].…”

Section: Discussionmentioning

confidence: 99%

“…These results are, in general, aligned with ours, as the results were overall positive, but also allowed to identify some aspects that need improvement in the future. Additionally, there are other contributions dedicated to the evaluation of commercially available voice assistants [ 16 , 67 , 70 ]. However, they do not perform the evaluation in the specific scenario of a smart home, considering mainly general commands related to the weather, shopping, alarms, directions, music, news, etc.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

An Accessible Smart Home Based on Integrated Multimodal Interaction

Rocha

Ketsmur

Almeida

et al. 2021

Sensors

View full text Add to dashboard Cite

Our homes are becoming increasingly sensorized and smarter. However, they are also becoming increasingly complex, making accessing them and their advantages difficult. Assistants have the potential for improving the accessibility of smart homes, by providing everyone with an integrated, natural, and multimodal way of interacting with the home’s ecosystem. To demonstrate this potential and contribute to more environmentally friendly homes, in the scope of the project Smart Green Homes, a home assistant highly integrated with an ICT (Information and communications technology) home infrastructure was developed, deployed in a demonstrator, and evaluated by seventy users. The users’ global impression of our home assistant is in general positive, with 61% of the participants rating it as good or excellent overall and 51% being likely or very likely to recommend it to others. Moreover, most think that the assistant enhances interaction with the smart home’s multiple devices and is easy to use by everyone. These results show that a home assistant providing an integrated view of a smart home, through natural, multimodal, and adaptive interaction, is a suitable solution for enhancing the accessibility of smart homes and thus contributing to a better living ambient for all of their inhabitants.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

An Accessible Smart Home Based on Integrated Multimodal Interaction

Rocha

Ketsmur

Almeida

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Evaluators reported that the heuristics enabled them to evaluate more thoroughly. Zwakman et al (2021) look at usability evaluations of AI-based voice assistants with an emphasis on Amazon Alexa ( Zwakman et al, 2021 ). The authors extend the use of System Usability Scale (SUS), often used in evaluating the usability of Graphical User Interfaces (GUI), to a Voice Usability Scale (VUS) for voice assistants.…”

Section: Literature Reviewmentioning

confidence: 99%

“…The authors extend the use of System Usability Scale (SUS), often used in evaluating the usability of Graphical User Interfaces (GUI), to a Voice Usability Scale (VUS) for voice assistants. Zwakman et al (2021) propose usability dimensions that align with many of the Hallmarks. They assess voice assistants across six usability dimensions: General usability (general thoughts about using the voice assistant), Affective (the psychological and feelings of the user), Recognition and visibility (the intuitiveness of the interactions with the voice assistant), Pragmatic (ability to accomplish tasks), Errors and frustration (prevention and handling of errors), and Guidance and help (guiding users on the use of the voice assistant to accomplish a task) ( Zwakman et al, 2021 ).…”

Section: Literature Reviewmentioning

confidence: 99%

“…Pleasant engagement is important for collaborative machine partners as it is for chatbots; chatbots have recently been assessed in user studies for dialogue properties such as repetitiveness, interestingness, engagingness, and humanness ( See et al, 2019 ). Ease-of-use is important for collaborative machine partners as well; usability heuristics have been created for user interfaces ( Nielsen and Molich 1990 ) and more recently voice-activated assistants ( Wei and Landay 2018 ; Zwakman et al, 2021 ). These usability heuristics serve as guidelines for development and are used in heuristic evaluation by usability experts to identify problems.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Assessing Open-Ended Human-Computer Collaboration Systems: Applying a Hallmarks Approach

Kozierok

Aberdeen

Clark

et al. 2021

Front. Artif. Intell.

View full text Add to dashboard Cite

There is a growing desire to create computer systems that can collaborate with humans on complex, open-ended activities. These activities typically have no set completion criteria and frequently involve multimodal communication, extensive world knowledge, creativity, and building structures or compositions through multiple steps. Because these systems differ from question and answer (Q&A) systems, chatbots, and simple task-oriented assistants, new methods for evaluating such collaborative computer systems are needed. Here, we present a set of criteria for evaluating these systems, called Hallmarks of Human-Machine Collaboration. The Hallmarks build on the success of heuristic evaluation used by the user interface community and past evaluation techniques used in the spoken language and chatbot communities. They consist of observable characteristics indicative of successful collaborative communication, grouped into eight high-level properties: robustness; habitability; mutual contribution of meaningful content; context-awareness; consistent human engagement; provision of rationale; use of elementary concepts to teach and learn new concepts; and successful collaboration. We present examples of how we used these Hallmarks in the DARPA Communicating with Computers (CwC) program to evaluate diverse activities, including story and music generation, interactive building with blocks, and exploration of molecular mechanisms in cancer. We used the Hallmarks as guides for developers and as diagnostics, assessing systems with the Hallmarks to identify strengths and opportunities for improvement using logs from user studies, surveying the human partner, third-party review of creative products, and direct tests. Informal feedback from CwC technology developers indicates that the use of the Hallmarks for program evaluation helped guide development. The Hallmarks also made it possible to identify areas of progress and major gaps in developing systems where the machine is an equal, creative partner.

show abstract

Exploring the Opinions of Experts in Conversational Design: A Study on Users’ Mental Models of Voice Assistants

Motta

Quaresma

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Usability Evaluation of Artificial Intelligence-Based Voice Assistants: The Case of Amazon Alexa

Cited by 50 publications

References 42 publications

An Accessible Smart Home Based on Integrated Multimodal Interaction

An Accessible Smart Home Based on Integrated Multimodal Interaction

Assessing Open-Ended Human-Computer Collaboration Systems: Applying a Hallmarks Approach

Exploring the Opinions of Experts in Conversational Design: A Study on Users’ Mental Models of Voice Assistants

Contact Info

Product

Resources

About