A statistical simulation technique to develop and evaluate conversational agents

Griol, David; Carbó, Javier; Molina, José M.

doi:10.3233/aic-130573

Cited by 16 publications

(7 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…There has been some recent work on simulating dialogues and computing metrics on top of these simulated dialogues [20,21]. However, this work would require using an agent to simulate the conversations and then using a metric on top of these simulated conversations -a process where metric computation is completely separated from dialogue generation.…”

Section: Related Workmentioning

confidence: 99%

How Am I Doing?: Evaluating Conversational Search Systems Offline

Lipani

Carterette

Yılmaz

2021

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

As conversational agents like Siri and Alexa gain in popularity and use, conversation is becoming a more and more important mode of interaction for search. Conversational search shares some features with traditional search, but differs in some important respects: conversational search systems are less likely to return ranked lists of results (a SERP), more likely to involve iterated interactions, and more likely to feature longer, well-formed user queries in the form of natural language questions. Because of these differences, traditional methods for search evaluation (such as the Cranfield paradigm) do not translate easily to conversational search. In this work, we propose a framework for offline evaluation of conversational search, which includes a methodology for creating test collections with relevance judgments, an evaluation measure based on a user interaction model, and an approach to collecting user interaction data to train the model. The framework is based on the idea of “subtopics”, often used to model novelty and diversity in search and recommendation, and the user model is similar to the geometric browsing model introduced by RBP and used in ERR. As far as we know, this is the first work to combine these ideas into a comprehensive framework for offline evaluation of conversational search.

show abstract

Section: Related Workmentioning

confidence: 99%

How Am I Doing?: Evaluating Conversational Search Systems Offline

Lipani

Carterette

Yılmaz

2021

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

show abstract

“…Secondly, we employed a statistical dialog simulation technique previously developed [73] to generate 50,000 successful dialogs for the task. The goal was to explore the space of possible dialog states to learn a new dialog model for the Demic metabot.…”

Section: Evaluation Of Our Proposalmentioning

confidence: 99%

Developing enhanced conversational agents for social virtual worlds

et al. 2019

Self Cite

View full text Add to dashboard Cite

In this paper, we present a methodology for the development of embodied conversational agents for social virtual worlds. The agents provide multimodal communication with their users in which speech interaction is included. Our proposal combines different techniques related to Artificial Intelligence, Natural Language Processing, Affective Computing, and User Modeling. Firstly, the developed conversational agents. A statistical methodology has been developed to model the system conversational behavior, which is learned from an initial corpus and improved with the knowledge acquired from the successive interactions. In addition, the selection of the next system response is adapted considering information stored into user's profiles and also the emotional contents detected in the user's utterances. Our proposal has been evaluated with the successful development of an embodied conversational agent which has been placed in the Second Life social virtual world. The avatar includes the different models and interacts with the users who inhabit the virtual world in order to provide academic information. The experimental results show that the agent's conversational behavior adapts successfully to the specific characteristics of users interacting in such environments.

show abstract

“…A previously developed statistical user simulation technique [67] has been used to complete a detailed evaluation of our proposal. The use of this kind of techniques has been already used in previous work to evaluate proposals to develop multidomain dialog systems [20,27,68,29,24].…”

Section: Overall Evaluation Of Our Proposalmentioning

confidence: 99%

Building multi-domain conversational systems from single domain resources

Griol

Molina

2018

Neurocomputing

Self Cite

View full text Add to dashboard Cite

Current advances in the development of mobile and smart devices have generated a growing demand for natural human-machine interaction and favored the intelligent assistant metaphor, in which a single interface gives access to a wide range of functionalities and services. Conversational systems constitute and important enabling technology in this paradigm. However, they are usually defined to interact in semanticrestricted domains in which users are offered a limited number of options and functionalities. The design of multi-domain systems implies that a single conversational system is able to assist the user in a variety of tasks. In this paper we propose an architecture for the development of multi-domain conversational systems that allows: (1) integrating available multi and single domain speech recognition and understanding modules, (2) combining available system in the different domains implied so that it is not necessary to generate new expensive resources for the multidomain system, (3) achieving better domain recognition rates to select the appropriate interaction management strategies. We have evaluated our proposal combining three systems in different domains to show that the proposed architecture can satisfactory deal with multimodain dialogs.

show abstract

A statistical simulation technique to develop and evaluate conversational agents

Cited by 16 publications

References 51 publications

How Am I Doing?: Evaluating Conversational Search Systems Offline

How Am I Doing?: Evaluating Conversational Search Systems Offline

Developing enhanced conversational agents for social virtual worlds

Building multi-domain conversational systems from single domain resources

Contact Info

Product

Resources

About