ConvAI Dataset of Topic-Oriented Human-to-Chatbot Dialogues

Logacheva, Varvara; Burtsev, Mikhail; Malykh, Valentin; Polulyakh, Vadim; Seliverstov, Aleksandr

doi:10.1007/978-3-319-94042-7_3

Cited by 14 publications

(11 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The dataset comprises 8,650 unique and unconstrained conversations conducted between June and October 2021 with English-speaking users in the US. With a total of 346,554 turns and an average of 44 turns per conversation, the dataset is almost twice the size of the existing human chat corpus Con-vAI (Logacheva et al, 2018). Further, with a ratio of 1.1 conversations per user, the corpus significantly exceeds the number of unique users, compared to similar previous studies (Völkel et al, 2021;Porcheron et al, 2018;Völkel et al, 2020).…”

Section: Datasetmentioning

confidence: 95%

Proceedings of the Second Workshop on Bridging Human--Computer Interaction and Natural Language Processing

2022

View full text Add to dashboard Cite

An existing domain taxonomy for normalizing content is often assumed when discussing approaches to information extraction, yet often in real-world scenarios there is none. When one does exist, as the information needs shift, it must be continually extended. This is a slow and tedious task, and one that does not scale well. Here we propose an interactive tool that allows a taxonomy to be built or extended rapidly and with a human in the loop to control precision. We apply insights from text summarization and information extraction to reduce the search space dramatically, then leverage modern pretrained language models to perform contextualized clustering of the remaining concepts to yield candidate nodes for the user to review. We show this allows a user to consider as many as 200 taxonomy concept candidates an hour to quickly build or extend a taxonomy to better fit information needs.

show abstract

Section: Datasetmentioning

confidence: 95%

Proceedings of the Second Workshop on Bridging Human--Computer Interaction and Natural Language Processing

2022

View full text Add to dashboard Cite

show abstract

“…Ideally, chatbots would be interactively evaluated, but due to the high cost, next utterance simulation is used as a surrogate. Although next utterance generation is a more artificial task, Logacheva et al (2018) observed a Pearson correlation of 0.6 between conversation-level and utterance-level ratings.…”

Section: Chatbot Evaluationmentioning

confidence: 99%

Item Response Theory for Efficient Human Evaluation of Chatbots

Sedoc¹,

Ungar²

2020

Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems

View full text Add to dashboard Cite

Conversational agent quality is currently assessed using human evaluation, and often requires an exorbitant number of comparisons to achieve statistical significance. In this paper, we introduce Item Response Theory (IRT) for chatbot evaluation, using a paired comparison in which annotators judge which system responds better to the next turn of a conversation. IRT is widely used in educational testing for simultaneously assessing the ability of test takers and the quality of test questions. It is similarly well suited for chatbot evaluation since it allows the assessment of both models and the prompts used to evaluate them. We use IRT to efficiently assess chatbots, and show that different examples from the evaluation set are better suited for comparing highquality (nearer to human performance) than low-quality systems. Finally, we use IRT to reduce the number of evaluation examples assessed by human annotators while retaining discriminative power.

show abstract

“…For simplicity, we refer to the conversation history of all chatbots nurtured on the LightBlue platform as the LightBlue Corpus. As shown in [65] 0.051 0.012 0.233 Twitter [66] 0.038 0.028 0.734 Cornell movie dialogues [67] 0.019 0.017 0.896…”

Section: Social Bondmentioning

confidence: 99%

LightBlue: Nurture Your Personal Chatbot

Zhang¹,

Liu²,

Gong³

et al. 2022

Artificial Intelligence Trends &Amp; Technologies

View full text Add to dashboard Cite

Chatbot has long been an important research topic in artificial intelligence and attracts lots of attention recently. Despite significant advancements in language ability, the interactions between users and chatbots are rather generic, short-term, and transnational. It has always been challenging to develop truly personal chatbots and even more difficult to establish longterm, affective connections. This paper first brings up “nurture” as a new interaction mode with chatbots. We introduce the nurture framework and accordingly design the learning algorithm and nurture functions. Then we present LightBlue – a platform that allows non-professionals to nurture personal chatbots from scratch. Experiments on both closed- and open-domain tasks validate the proposed framework and demonstrate a promising method for facilitating long-term interaction between users and chatbots.

show abstract

ConvAI Dataset of Topic-Oriented Human-to-Chatbot Dialogues

Cited by 14 publications

References 2 publications

Proceedings of the Second Workshop on Bridging Human--Computer Interaction and Natural Language Processing

Proceedings of the Second Workshop on Bridging Human--Computer Interaction and Natural Language Processing

Item Response Theory for Efficient Human Evaluation of Chatbots

LightBlue: Nurture Your Personal Chatbot

Contact Info

Product

Resources

About