Predicting User Satisfaction with Intelligent Assistants

Kiseleva, Julia; Williams, Kyle; Awadallah, Ahmed Hassan; Crook, Aidan C.; Zitouni, Imed; Anastasakos, Tasos

doi:10.1145/2911451.2911521

Cited by 93 publications

(60 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To evaluate a search system, satisfaction can be considered as regarding not only to the whole search experience but also to some speci c aspects [46], such as the precision or completeness of search results, response time and so on. Since satisfaction is important for both search engine evaluation and optimization, a number of research studies have tried to quantify user satisfaction in both desktop search [20,47] and mobile search [26,28], and in both homogeneous [31] and heterogeneous search environment [10]. In recent years, a number of works (e.g, [21,22]) have started using the bene t-cost framework to analyze the satisfaction judgement process of users.…”

Section: Search Satisfactionmentioning

confidence: 99%

Meta-evaluation of Online and Offline Web Search Evaluation Metrics

Chen

Zhou

Liu

et al. 2017

Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

As in most information retrieval (IR) studies, evaluation plays an essential part in Web search research. Both o ine and online evaluation metrics are adopted in measuring the performance of search engines. O ine metrics are usually based on relevance judgments of query-document pairs from assessors while online metrics exploit the user behavior data, such as clicks, collected from search engines to compare search algorithms. Although both types of IR evaluation metrics have achieved success, to what extent can they predict user satisfaction still remains under-investigated. To shed light on this research question, we meta-evaluate a series of existing online and o ine metrics to study how well they infer actual search user satisfaction in di erent search scenarios. We nd that both types of evaluation metrics signi cantly correlate with user satisfaction while they re ect satisfaction from di erent perspectives for di erent search tasks. O ine metrics be er align with user satisfaction in homogeneous search (i.e. ten blue links) whereas online metrics outperform when vertical results are federated. Finally, we also propose to incorporate mouse hover information into existing online evaluation metrics, and empirically show that they be er align with search user satisfaction than click-based online metrics.

show abstract

Section: Search Satisfactionmentioning

confidence: 99%

Meta-evaluation of Online and Offline Web Search Evaluation Metrics

Chen

Zhou

Liu

et al. 2017

Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…Until recently, conversational systems have focused on completing a task, i.e., booking a flight, providing automotive customer support, or describing a restaurant [16,18,39,57,59,60]. Much recent work on conversational AI systems also presupposes a specific "information need" [10,23,40]. These types of dialogue systems have very different objectives from our goal of creating a casual open-domain social conversational system.…”

Section: Related Workmentioning

confidence: 99%

Entertaining and opinionated but too controlling

Bowden

Cui

et al. 2019

Proceedings of the 1st International Conference on Conversational User Interfaces

View full text Add to dashboard Cite

Conversational systems typically focus on functional tasks such as scheduling appointments or creating todo lists. Instead we design and evaluate SlugBot (SB), one of 8 semifinalists in the 2018 Alexa Prize, whose goal is to support casual open-domain social interaction. This novel application requires both broad topic coverage and engaging interactive skills. We developed a new technical approach to meet this demanding situation by crowd-sourcing novel content and introducing playful conversational strategies based on storytelling and games. We collected over 10,000 conversations during August 2018 as part of the Alexa Prize competition. We also conducted an in-lab follow-up qualitative evaluation. Overall users found SB moderately engaging; conversations averaged 3.6 minutes and involved 26 user turns. However, users reacted very differently to different conversation subtypes. Storytelling and games were evaluated positively; these were seen as entertaining with predictable interactive structure. They also led users to impute personality and intelligence to SB. In contrast, search and general Chit-Chat induced coverage problems; here users found it hard to infer what topics SB could understand, with these conversations seen as being too system-driven. Theoretical and design implications suggest a move away from conversational systems that simply provide factual information. Future systems should be designed to have their own opinions with personal stories to share, and SB provides an example of how we might achieve this.

show abstract

“…Julia Kiseleva et al have done a continuous and in-depth study on the user satisfaction of intelligent assistant, and found that the factors affecting task satisfaction are different in different scenarios. They also try to use interactive signals to construct a method to predict the satisfaction of intelligent assistants [8,9,10]. Other studies have found that there is a relationship between the personality of users and their preference for different intelligent assistants [11].…”

Section: Related Workmentioning

confidence: 99%

Visual Feedback Design and Research of Handheld Mobile Voice Assistant Interface

Yang¹

2018

Proceedings of the 2018 2nd International Conference on Artificial Intelligence: Technologies and Applications (ICAITA 2018)

View full text Add to dashboard Cite

Abstract-Based on the limitation of handheld mobile voice assistant in usability, this paper analyzes the important role and characteristics of visual feedback in voice interaction interface, and discusses the research and the application status of voice assistant usability. On this basis, taking the interaction of Siri voice assistant as a case to study, the task characteristics and design defects of voice interaction are found through interview 12 users. Based on the general design principles and voice interaction design principles, the voice interaction feedback interface in handheld mobile devices is optimized to improve the validity and reliability of the intelligent voice assistant.

show abstract

Predicting User Satisfaction with Intelligent Assistants

Cited by 93 publications

References 46 publications

Meta-evaluation of Online and Offline Web Search Evaluation Metrics

Meta-evaluation of Online and Offline Web Search Evaluation Metrics

Entertaining and opinionated but too controlling

Visual Feedback Design and Research of Handheld Mobile Voice Assistant Interface

Contact Info

Product

Resources

About