Heuristic Evaluation of Conversational Agents

Langevin, Raina; Lordon, Ross J.; Avrahami, Thi; Cowan, Benjamin R.; Hirsch, Tad; Hsieh, Gary

doi:10.1145/3411764.3445312

Cited by 65 publications

(34 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The need to update heuristics for new technology is evident in recent studies that utilized custom usability principles to evaluate smartwatches [13]. Recently, researchers also proposed a set of adapted heuristics for conversational agents [53] and for VR products [75], based on testing and feedback by experts. Future work should develop sets of reusable heuristics that apply to new categories of products.…”

Section: Impact Of Emerging Technologies On Ux Practicesmentioning

confidence: 99%

“Merging Results Is No Easy Task”: An International Survey Study of Collaborative Data Analysis Practices Among UX Practitioners

Kuang

Jin

Fan

2022

CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Analysis is a key part of usability testing where UX practitioners seek to identify usability problems and generate redesign suggestions. Although previous research reported how analysis was conducted, the findings were typically focused on individual analysis or based on a small number of professionals in specific geographic regions. We conducted an online international survey of 279 UX practitioners on their practices and challenges while collaborating during data analysis. We found that UX practitioners were often under time pressure to conduct analysis and adopted three modes of collaboration: independently analyze different portions of the data and then collaborate, collaboratively analyze the session with little or no independent analysis, and independently analyze the same set of data and then collaborate. Moreover, most encountered challenges related to lack of resources, disagreements with colleagues regarding usability problems, and difficulty merging analysis from multiple practitioners. We discuss design implications to better support collaborative data analysis. CCS CONCEPTS• Human-centered computing → Empirical studies in collaborative and social computing.

show abstract

Section: Impact Of Emerging Technologies On Ux Practicesmentioning

confidence: 99%

“Merging Results Is No Easy Task”: An International Survey Study of Collaborative Data Analysis Practices Among UX Practitioners

Kuang

Jin

Fan

2022

CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…The Gricean Maxims are a collection of communication principles, to which both speaker and listener should adhere to engage in effective communication. Researchers have been using Gricean Maxims to evaluate both human-human conversations (Eskritt et al, 2008;Kleinke, 2010) and human-agent conversations (Xiao et al, 2020;Langevin et al, 2021). In the context of conducting conversational surveys, a "cooperative" interviewer would obey all the maxims to form questions and probe quality responses.…”

Section: Gricean Scoresmentioning

confidence: 99%

What should I Ask: A Knowledge-driven Approach for Follow-up Questions Generation in Conversational Surveys

Xiao¹,

Diesner²,

Ji³

et al. 2022

Preprint

View full text Add to dashboard Cite

Conversational surveys, where an agent asks open-ended questions through natural language interfaces, offer a new way to collect information from people. A good follow-up question in a conversational survey prompts high-quality information and delivers engaging experiences. However, generating highquality follow-up questions on the fly is a nontrivial task. The agent needs to understand the diverse and complex participant responses, adhere to the survey goal, and generate clear and coherent questions. In this study, we propose a knowledge-driven follow-up question generation framework. The framework combines a knowledge selection module to identify salient topics in participants' responses and a generative model guided by selected knowledge entity-relation pairs. To investigate the effectiveness of the proposed framework, we build a new dataset for open-domain follow-up question generation and present a new set of reference-free evaluation metrics based on Gricean Maxim. Our experiments demonstrate that our framework outperforms a GPT-based baseline in both objective evaluation and human-expert evaluation.

show abstract

“…Grounded in Conversational Analysis, Moore and Arar [32] suggested borrowing principles of how people design their dialogues with considering the three characteristics: Recipient Design, Minimization, and Repair. Drawing inspiration from Grice's Cooperative Principle in pragmatic theory and Nielsen's usability heuristics for user interface design, Langevin et al [22] developed a set of usability heuristics to guide and evaluate the design of conversational agents. Wei and Landay [39] also suggested that the system "[a]dapt agent style to who users are, how they speak, and how they are feeling" in the heuristics.…”

Section: Related Work 21 Conversation Design For Voice Interfacesmentioning

confidence: 99%

“…Grice's Maxims describe "how" as the manner dimension of cooperation, which emphasizes clarity, brevity, and order with the goal of effective information exchange. The Cooperative Principle, which includes manner, has been widely adopted by researchers and designers to design voice interfaces where brevity is encouraged to achieve the maximum efficiency [22]. On the other hand, Brown and Levinson [3] proposed Politeness Theory as a deviation from Grice's Maxims.…”

Section: Linguistic Strategiesmentioning

confidence: 99%

“…However, the design space for these behaviors involves competing objectives with significant design tradeoffs. For example, on the one hand, a major focus of conversational design guidelines proposed in prior research is the efficiency of interaction [22]. Researchers have suggested that efficiency can be achieved by maintaining minimal information exchange in short utterances and making progress in the dialogue through follow-up questions [8,32,39].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Polite or Direct? Conversation Design of a Smart Display for Older Adults Based on Politeness Theory

Qu¹,

Maus²,

Mutlu³

2022

Preprint

View full text Add to dashboard Cite

In this paper, we applied politeness strategies to seven speech acts of a smart display, identified four personas representing older adults' experience of system usage and preferences for politeness, and suggested personalized design strategies for each persona. (A) An older adult using the conversational interface; (B) an example of polite speech used by our interface; and (C) an example of direct speech used by our interface.

show abstract

Heuristic Evaluation of Conversational Agents

Cited by 65 publications

References 17 publications

“Merging Results Is No Easy Task”: An International Survey Study of Collaborative Data Analysis Practices Among UX Practitioners

“Merging Results Is No Easy Task”: An International Survey Study of Collaborative Data Analysis Practices Among UX Practitioners

What should I Ask: A Knowledge-driven Approach for Follow-up Questions Generation in Conversational Surveys

Polite or Direct? Conversation Design of a Smart Display for Older Adults Based on Politeness Theory

Contact Info

Product

Resources

About