Evaluating and Informing the Design of Chatbots

Jain, Mohit; Kumar, Pratyush; Kota, Ramachandra; Patel, Shwetak

doi:10.1145/3196709.3196735

Cited by 234 publications

(191 citation statements)

References 18 publications

Supporting

Mentioning

177

Contrasting

Unclassified

Order By: Relevance

“…Chaves and Gerosa (2019) describe intelligence as the ability of a chatbot to participate in a dialogue with an awareness of the discussed topic, while Jain et al (2018) believe that the intelligence of a chatbot can be also deduced from its ability to proactively ask suitable questions and to involve the participant in a meaningful and human-like dialogue. At a holistic level, in line with the proposed final taxonomy, the design elements for chatbots related to specific intelligence features can be described using 15 characteristics, which in turn can be categorized into the following 5 dimensions: The intelligence framework D 1 depicts the underlying cognitive system design delimiting the technical principles under which a chatbot communicates, processes information, and/or selects an action or response (Saravanan et al 2017;Knote et al 2018;Diederich et al 2019).…”

Section: Intelligencementioning

confidence: 99%

“…Chatbots as a form of conversational agents have been developed for different applications. This is due to the evolving of artificial intelligence (AI) and natural language processing (NLP), which is changing the way artificial assistants communicate and interact with people (Nguyen and Sidorova 2018;Jain et al 2018). By improving text-tospeech and speech-to-text communication, the use of chatbots has become more convenient and common (Bittner et al 2019).…”

Section: Introductionmentioning

confidence: 99%

“…For instance, new smart assistants, such as Cortana, Alexa, Google Assistant and Siri, have been designed with the intention of supporting users in everyday life as voice-activated intelligent personal assistants. The proliferation of these assistants has contributed to the popularity of chatbots worldwide (Di Prospero et al 2017;Gnewuch et al 2017;Jain et al 2018;Diederich et al 2019). This in turn has also led to an increasing use of domain-specific chatbots (Di Prospero et al 2017).…”

Section: Introductionmentioning

confidence: 99%

“…Although some elemental chatbot classification frameworks can be found in scientific literature, the research is dispersed into different thematic axes and research areas. Furthermore, the scientific and practical knowledge about chatbots has also grown in a segregated manner given a shortage of integrative perspectives to support chatbot development and design processes Jain et al 2018;Piccolo et al 2018). For instance, most scientific studies today concentrate on particular aspects of chatbots, such as the personality of cognitive chatbots, technical capabilities or their specific application Accepted after three revisions by the editors of the Special Issue.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Virtual Assistance in Any Context

et al. 2020

View full text Add to dashboard Cite

Several domain-specific assistants in the form of chatbots have conquered many commercial and private areas. However, there is still a limited level of systematic knowledge of the distinctive characteristics of design elements for chatbots to facilitate development, adoption, implementation, and further research. To close this gap, the paper outlines a taxonomy of design elements for chatbots with 17 dimensions organized into the perspectives intelligence, interaction and context. The conceptually grounded design elements of the taxonomy are used to analyze 103 chatbots from 23 different application domains. Through a clustering-based approach, five chatbot archetypes that currently exist for domain-specific chatbots are identified. The developed taxonomy provides a structure to differentiate and categorize domain-specific chatbots according to archetypal qualities that guide practitioners when taking design decisions. Moreover, the taxonomy serves academics as a foundation for conducting further research on chatbot design while integrating scientific and practical knowledge.Keywords Chatbot taxonomy Á Design elements Á Domain-specific chatbots Á Human computer interaction Bus Inf Syst Eng 62(3): 211-225 (2020)

show abstract

Section: Intelligencementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Virtual Assistance in Any Context

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Chatbots, or "text messaging-based conversational agents", have received particular attention in 2010s (Jain et al, 2018). Many modern text-based chatbots use relatively simple NLP tools (Abdul-Kader and Woods, 2015), or avoid ML/NLP altogether (Pereira and Díaz, 2018), relying on conver-sation flow design and non-NLP inputs like buttons and quick-replies.…”

Section: Introductionmentioning

confidence: 99%

Evaluation and Improvement of Chatbot Text Classification Data Quality Using Plausible Negative Examples

Kuksenok¹,

Martyniv²

2019

Proceedings of the First Workshop on NLP for Conversational AI

View full text Add to dashboard Cite

We describe and validate a metric for estimating multi-class classifier performance based on cross-validation and adapted for improvement of small, unbalanced natural-language datasets used in chatbot design. Our experiences draw upon building recruitment chatbots that mediate communication between job-seekers and recruiters by exposing the ML/NLP dataset to the recruiting team. Evaluation approaches must be understandable to various stakeholders, and useful for improving chatbot performance. The metric, nex-cv, uses negative examples in the evaluation of text classification, and fulfils three requirements. First, it is actionable: it can be used by non-developer staff. Second, it is not overly optimistic compared to human ratings, making it a fast method for comparing classifiers. Third, it allows model-agnostic comparison, making it useful for comparing systems despite implementation differences. We validate the metric based on seven recruitmentdomain datasets in English and German over the course of one year.

show abstract

Artificial intelligence changes the way we work: A close look at innovating with chatbots

Wang

Lin

Shao

2022

Asso for Info Science & Tech

View full text Add to dashboard Cite

An enhanced understanding of the innovative use of artificial intelligence (AI) is essential for organizations to improve work design and daily business operations. This study's purpose is to offer insights into how AI can transform organizations' work practices through diving deeply into its innovative use in the context of a primary AI tool, a chatbot, and examining the antecedents of innovative use by conceptualizing employee trust as a multidimensional construct and exploring employees' perceived benefits. In particular, we have conceptualized employee trust in chatbots as a second‐order construct, including three first‐order variables: trust in functionality, trust in reliability, and trust in data protection. We collected data from 202 employees. The results supported our conceptualization of trust in chatbots and showed that three dimensions of first‐order trust beliefs have relatively the same level of importance. Further, both knowledge support and work–life balance enhance trust in chatbots, which in turn leads to innovative use of chatbots. Our study contributes to the existing literature by introducing the new conceptualization of trust in chatbots and examining its antecedents and outcomes. The results can provide important practical insights regarding how to support innovative use of chatbots as the new way we organize work.

show abstract

Evaluating and Informing the Design of Chatbots

Cited by 234 publications

References 18 publications

Virtual Assistance in Any Context

Virtual Assistance in Any Context

Evaluation and Improvement of Chatbot Text Classification Data Quality Using Plausible Negative Examples

Artificial intelligence changes the way we work: A close look at innovating with chatbots

Contact Info

Product

Resources

About