Effects of Communication Directionality and AI Agent Differences in Human-AI Interaction

Ashktorab, Zahra; Dugan, Casey; Johnson, James M.; Qian, Pan; Zhang, Wei; Kumaravel, Sadhana; Campbell, Murray

doi:10.1145/3411764.3445256

Cited by 23 publications

(11 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is a growing trend in the usage of conversational agents in our daily lives, and conversation is becoming a key mode of human-computer interaction [49]. Prior research investigated the use of conversational agents in a variety of contexts (e.g., collaborative games [1,3], customer services [2], journaling and reflection [42], productivity applications [23], and business documents [34]). However, the use of a conversational assistant for UX analysis has been unexplored.…”

Section: Human-ai Collaboration Via Interactive Conversational Assist...mentioning

confidence: 99%

Collaboration with Conversational AI Assistants for UX Evaluation: Questions and How to Ask them (Voice vs. Text)

Kuang

Soure

Fan

et al. 2023

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

AI is promising in assisting UX evaluators with analyzing usability tests, but its judgments are typically presented as non-interactive visualizations. Evaluators may have questions about test recordings, but have no way of asking them. Interactive conversational assistants provide a Q&A dynamic that may improve analysis efficiency and evaluator autonomy. To understand the full range of analysis-related questions, we conducted a Wizard-of-Oz design probe study with 20 participants who interacted with simulated AI assistants via text or voice. We found that participants asked for five categories of information: user actions, user mental model, help from the AI assistant, product and task information, and user demographics. Those who used the text assistant asked more questions, but the question lengths were similar. The text assistant was perceived as significantly more efficient, but both were rated equally in satisfaction and trust. We also provide design considerations for future conversational AI assistants for UX evaluation.CCS Concepts: • Human-centered computing → Empirical studies in collaborative and social computing; Natural language interfaces.

show abstract

Section: Human-ai Collaboration Via Interactive Conversational Assist...mentioning

confidence: 99%

Collaboration with Conversational AI Assistants for UX Evaluation: Questions and How to Ask them (Voice vs. Text)

Kuang

Soure

Fan

et al. 2023

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…Researchers have investigated user perceptions of AI in different domains [1,33,39], since social perception of one's partner in a collaborative space can impact the outcome of the collaboration. The perceived interactivity -or lack thereof -of systems can have an impact on user perceptions of the system [39] as most existing co-creative systems use one-way communication.…”

Section: Communication In Human-ai Co-creationmentioning

confidence: 99%

Understanding User Perceptions, Collaborative Experience and User Engagement in Different Human-AI Interaction Designs for Co-Creative Systems

Rezwana,

Maher

2022

Preprint

View full text Add to dashboard Cite

Human-AI co-creativity involves humans and AI collaborating on a shared creative product as partners. In a creative collaboration, communication is an essential component among collaborators. In many existing co-creative systems users can communicate with the AI, usually using buttons or sliders. Typically, the AI in co-creative systems cannot communicate back to humans, limiting their potential to be perceived as partners rather than just a tool. This paper presents a study with 38 participants to explore the impact of two interaction designs, with and without AI-to-human communication, on user engagement, collaborative experience and user perception of a co-creative AI. The study involves user interaction with two prototypes of a co-creative system that contributes sketches as design inspirations during a design task. The results show improved collaborative experience and user engagement with the system incorporating AI-to-human communication. Users perceive co-creative AI as more reliable, personal, and intelligent when the AI communicates to users. The findings can be used to design effective co-creative systems, and the insights can be transferred to other fields involving human-AI interaction and collaboration.CCS Concepts: • Human-centered computing → Empirical studies in interaction design.

show abstract

“…Second, their usage of ConceptNet is different from our usage of BabelNet because they use vector representations derived from an ensemble of word2vec, GloVe, and ConceptNet using retrofitting (Speer, Chin, & Havasi, 2017) whereas we leverage the graph structure of BabelNet. Ashktorab et al (2021) propose three different AI approaches for a word game similar to Taboo, including a supervised model trained on Taboo card words, a reinforcement learning model, and count-based model using a word evocation dataset. Their task, in which an agent gives clues until a user guesses the secret word, is different from Codenames.…”

Section: Related Workmentioning

confidence: 99%

“…However, the datasets used in their models present interesting directions for future work in the Codenames task. Ashktorab et al (2021) propose three different AI approaches for a word game similar to Taboo, including a supervised model trained on Taboo card words, a reinforcement learning model, and count-based model using the Small World of Words (De Deyne et al, 2019), a word evocation dataset. Their task, in which an agent gives clues until a user guesses the secret word, is different from Codenames.…”

Section: Related Workmentioning

confidence: 99%

Playing Codenames with Language Graphs and Word Embeddings

Koyyalagunta¹,

Sun²,

Draelos³

et al. 2021

Preprint

View full text Add to dashboard Cite

Although board games and video games have been studied for decades in artificial intelligence research, challenging word games remain relatively unexplored. Word games are not as constrained as games like chess or poker. Instead, word game strategy is defined by the players' understanding of the way words relate to each other. The word game Codenames provides a unique opportunity to investigate common sense understanding of relationships between words, an important open challenge. We propose an algorithm that can generate Codenames clues from the language graph BabelNet or from any of several embedding methods -word2vec, GloVe, fastText or BERT. We introduce a new scoring function that measures the quality of clues, and we propose a weighting term called DETECT that incorporates dictionary-based word representations and document frequency to improve clue selection. We develop BabelNet-Word Selection Framework (BabelNet-WSF) to improve BabelNet clue quality and overcome the computational barriers that previously prevented leveraging language graphs for Codenames. Extensive experiments with human evaluators demonstrate that our proposed innovations yield state-of-the-art performance, with up to 102.8% improvement in precision@2 in some cases. Overall, this work advances the formal study of word games and approaches for common sense language understanding.

show abstract

Effects of Communication Directionality and AI Agent Differences in Human-AI Interaction

Cited by 23 publications

References 45 publications

Collaboration with Conversational AI Assistants for UX Evaluation: Questions and How to Ask them (Voice vs. Text)

Collaboration with Conversational AI Assistants for UX Evaluation: Questions and How to Ask them (Voice vs. Text)

Understanding User Perceptions, Collaborative Experience and User Engagement in Different Human-AI Interaction Designs for Co-Creative Systems

Playing Codenames with Language Graphs and Word Embeddings

Contact Info

Product

Resources

About