Usability guidelines and evaluation criteria for conversational user interfaces

Sugisaki, Kyoko; Bleiker, Andreas

doi:10.1145/3404983.3405505

Cited by 20 publications

(12 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As an example of work that presents an evaluation method based on checklist inspection, we can mention Sugisaki and Bleiker [2020], which presents a technology composed of 53 useful verification items for evaluating conversational interfaces based on Nielsen's heuristics . The proposed technology was submitted for evaluation by 15 professionals who analyzed each item and answered a questionnaire about the relevance of each one and how efficient, pleasant, convenient and effective the checklist was.…”

Section: Current Status and Related Workmentioning

confidence: 99%

Proposing Usability-UX technologies for the design and evaluation of text-based chatbots

Mafra,

Nunes,

Rocha

et al. 2024

JIS

View full text Add to dashboard Cite

Chatbots are interactive systems that communicate using natural language with human users, via a textual interface or voice activation. These tools are useful for many spheres of business such as Customer Service, Sales, Education and Learning, Health and Entertainment. Recently, chatbots have become popular, with significant growth in the software industry, especially text-based chatbots. This is encouraging developers to create their own tools, as well as attracting efforts from researchers into this area. Despite this highlight, technologies to guarantee the quality of chatbots and user satisfaction are not keeping up with the growing demand for these tools. Considering this, there is a need to propose technologies capable of supporting developers and development teams in the process of building and evaluating chatbots. Therefore, this research proposes to develop artifacts applicable to the design and evaluation process of chatbots, based on quality attributes identified in systematic literature reviews related to Usability and User Experience (UX), due to the importance and impact that these aspects have on user satisfaction and the perceived quality of the system. The first artifact is the U2Chatbot inspection checklist, developed to assist development teams in the process of identifying defects in text-based chatbots. The second artifact is a set of interface design patterns, DP-U2Chatbot, containing useful examples to support developers in the process of building chatbots. The technologies were subjected to the necessary evaluations. The results of the empirical study regarding the U2Chatbot inspection checklist indicated that participants considered the technology useful for discovering defects in chatbots, however, ease of use could be improved. The participants' experience discreetly influenced the effectiveness and efficiency of the technique, leading us to believe that professionals with a certain level of inspection experience can benefit more from the checklist. Regarding the evaluation of DP-U2Chatbot design patterns, the results generally indicated that the technology is easy to understand and useful in supporting the design of chatbots, helping to build better tools.

show abstract

Section: Current Status and Related Workmentioning

confidence: 99%

Proposing Usability-UX technologies for the design and evaluation of text-based chatbots

Mafra,

Nunes,

Rocha

et al. 2024

JIS

View full text Add to dashboard Cite

show abstract

“…Natural Language Processing (NLP), a branch of AI, enhances both the UI and UX by enabling more intuitive and human-like interactions. While traditional usability heuristics offer some guidance for UI design, they often fall short in the realm of conversational interfaces (Sugisaki & Bleiker, 2020). For these interfaces, specialized heuristics and metrics have been proposed to evaluate their effectiveness and usability (Følstad & Skjuve, 2019).…”

Section: User Interface and User Experience In Ai-driven Chatbot Deve...mentioning

confidence: 99%

The Development of Chatbot Provided Registration Information Services for Students in Distance Learning

Chumkaew

2023

ABAC J

View full text Add to dashboard Cite

In recent years, chatbots have become crucial, particularly for assisting students with real-time registration information. This research focused on 1) synthesizing registry works related to information provided for students, 2) designing chatbots and conversation structures in the form of interactive conversations between students and robots for answering questions and providing information tailored to their needs, and 3) examining and evaluating the use of chatbots in providing information services to students, while analyzing the accuracy and suitability of the developed chatbot. This study, based on research and development, utilized a sample consisting of 16 staff directly involved in the provision of registration information to students and 255 undergraduate students from Sukhothai Thammathirat Open University, with respondents being selected through a simple random sampling technique. The synthesis of the research results revealed the following findings: 1) A qualitative study revealed that the registration information related to students, called STOU Journey, consisted of 10 issues, and was required for the whole learning period. 2) The result of the design and development of the chatbot revealed that the developer chatbot could be used on both the website and the LINE application. It was also found that the chatbot could answer most questions correctly and completely. The chatbot responded quickly and was easy to use. The chatbot used language that was easy to understand and natural, while 3) satisfactory evaluation results from 255 undergraduate students showed that overall, students who had used the completed version of the chatbot were satisfied with the use of the chatbot at a high level (Mean = 4.19, SD = 0.98) while they also felt that the chatbot was easy to use (Mean = 4.33, SD = 0.95) and the using the chatbot felt like a natural conversation (Mean = 4.22, SD = 0.99).

show abstract

“…Chatbots are still an emerging technology, making the classification complex and multi-layered (Følstad et al 2019). Chatbots can be classified according to different criteria and categories, including the interaction type (voice-or textbased), the knowledge area of the chatbot/its field of application (domain-specific or open domain), the goal or use case of chatbots (task-oriented or non-task-oriented), and the design approaches/techniques used in the development of chatbots (rule-based or conversational AI) (Følstad et al 2019;Hussain et al 2019;Ramesh et al 2017;Sugisaki and Bleiker 2020).…”

Section: Chatbot Classificationsmentioning

confidence: 99%

“…In this paper, we follow definitions by Sugisaki and Bleiker (2020) and define writing-based chatbots as chatbots that allow users to freely express their utterances (via text) and initiate a conversation through the use of conversational AI. Click-based chatbots, on the other hand, are defined as chatbots that enable interaction with the user via predefined answers of the chatbot, which the user can click on.…”

Section: Chatbot Classificationsmentioning

confidence: 99%

“Clicking vs. Writing”—The Impact of a Chatbot’s Interaction Method on the Working Alliance in AI-based Coaching

Mai

Neef

Richert

2022

Coaching Theor. Prax.

View full text Add to dashboard Cite

Coaching has become an important didactic tool for reflecting learning processes in higher education. Digital media and AI-based technologies such as chatbots can support stimulating self-coaching processes. For the use case of student coaching on the topic of exam anxiety, the working alliance between a coaching chatbot and a human coachee is investigated. Two coachbot interaction methods are compared: A click-based chatbot (implemented in a rule-based system), where the coachee can only click on one answer, and a writing-based chatbot (implemented in a conversational AI), which allows the coachee to freely type in their answers. The focus is on which coachbot interaction method enables a stronger working alliance between coach and coachee: a click-based or a writing-based chatbot. The working alliance and the technical realization of the chatbot systems were investigated in an exploratory quantitative study with 21 engineering students. The results indicate that the working alliance in both study conditions can be classified as medium to high overall. The results further show higher values for bonding on a writing-based platform than when using a click-based system. However, click-based systems seem to be more helpful as a low-threshold entry point to coaching, as they guide coachees better through the process by providing predefined answers. An evaluation of the technical realization shows that self-reflection processes through digital self-coaching via chatbot are generally well accepted by students. For further development and research, it is therefore recommendable to develop a “mixed” coachbot that allows interaction via clicking as well as via free writing.

show abstract

Usability guidelines and evaluation criteria for conversational user interfaces

Cited by 20 publications

References 45 publications

Proposing Usability-UX technologies for the design and evaluation of text-based chatbots

Proposing Usability-UX technologies for the design and evaluation of text-based chatbots

The Development of Chatbot Provided Registration Information Services for Students in Distance Learning

“Clicking vs. Writing”—The Impact of a Chatbot’s Interaction Method on the Working Alliance in AI-based Coaching

Contact Info

Product

Resources

About