Relations between de-facto criteria in the evaluation of a spoken dialogue system

Abstract. In this paper, we describe a conversational agent which provides academic information. The dialog model of this agent has been developed by means of a statistical methodology that automatically explores the dialog space and allows learning new enhanced dialog strategies from a dialog corpus. A dialog simulation technique has been applied to acquire data required to train the dialog model and then explore the new dialog strategies. A set of measures has also been defined to evaluate the dialog strategy. The results of the evaluation show how the dialog model deviates from the initially predefined strategy, allowing the conversational agent to tackle new situations and generate new coherent answers for the situations already present in the initial corpus. The proposed technique can be used not only to develop new dialog managers but also to explore new enhanced dialog strategies focused on user adaptation required to interact in AmI environments.

show abstract

Section: Design Of An Academic Conversational Agentmentioning

confidence: 99%

“…A set of 100 dialogs was acquired with the conversational agent by means of its interaction with real users using an initial version of the system [1]. The acquisition process resulted in a spontaneous Spanish speech dialog corpus with 60 different speakers.…”

Section: Design Of An Academic Conversational Agentmentioning

confidence: 99%

Optimizing Dialog Strategies for Conversational Agents Interacting in AmI Environments

Griol

Carbó

Molina

2012

Ambient Intelligence - Software and Applications

View full text Add to dashboard Cite

show abstract

“…The design of our conversational agent is based on the requirements defined for a dialog system developed to provide spoken access to academic information about the Department of Languages and Computer Systems in the University of Granada [1]. To successfully manage the interaction with the users, the conversational agent carry out six main tasks: automatic speech recognition (ASR), natural language understanding (NLU), dialog management (DM), database access and storage (DB), natural language generation (NLG), and text-to-speech synthesis (TTS).…”

Section: Design Of An Academic Conversational Agentmentioning

confidence: 99%

“…• Task-independent A set of 100 dialogs was acquired with the conversational agent by means of its interaction with real users using an initial version of the system [1]. The acquisition process resulted in a spontaneous Spanish speech dialog corpus with 60 different speakers.…”

Section: Design Of An Academic Conversational Agentmentioning

confidence: 99%

Agent Simulation to Develop Interactive and User-Centered Conversational Agents

Griol

Carbó

Molina

2011

Advances in Intelligent and Soft Computing

View full text Add to dashboard Cite

Abstract. In this paper, we present a technique for developing user simulators which are able to interact and evaluate conversational agents. Our technique is based on a statistical model that is automatically learned from a dialog corpus. This model is used by the user simulator to provide the following answer taking into account the complete history of the interaction. The main objective of our proposal is not only to evaluate the conversational agent, but also to improve this agent by employing the simulated dialogs to learn a better dialog model. We have applied this technique to design and evaluate a conversational agent which provides academic information in a multi-agent system. The results of the evaluation show that the conversational agent reduces the time needed to fulfill to complete the the dialogs, thereby allowing the conversational agent to tackle new situations and generate new coherent answers for the situations already present in an initial model.

show abstract

“…However there are not any standard for assessing spoken dialogue systems, despite different de-facto standards are commonly used (Callejas and López-Cózar, 2008). Furthermore, the definition of metrics to compare the relevance of different contextual information sources in a user-based speech system is still under development.…”

Section: Introductionmentioning

confidence: 99%

EVALUATION OF A USER-ADAPTED SPOKEN LANGUAGE DIALOGUE SYSTEM - Measuring the Relevance of the Contextual Information Sources

Lucas-Cuesta

Fernández-Martínez²,

Rada³

et al. 2011

Proceedings of the 3rd International Conference on Agents and Artificial Intelligence

View full text Add to dashboard Cite

Abstract:We present an evaluation of a spoken language dialogue system with a module for the management of userrelated information, stored as user preferences and privileges. The flexibility of our dialogue management approach, based on Bayesian Networks (BN), together with a contextual information module, which performs different strategies for handling such information, allows us to include user information as a new level into the Context Manager hierarchy. We propose a set of objective and subjective metrics to measure the relevance of the different contextual information sources. The analysis of our evaluation scenarios shows that the relevance of the short-term information (i.e. the system status) remains pretty stable throughout the dialogue, whereas the dialogue history and the user profile (i.e. the middle-term and the long-term information, respectively) play a complementary role, evolving their usefulness as the dialogue evolves.

show abstract

Relations between de-facto criteria in the evaluation of a spoken dialogue system

Cited by 30 publications

References 32 publications

Optimizing Dialog Strategies for Conversational Agents Interacting in AmI Environments

Optimizing Dialog Strategies for Conversational Agents Interacting in AmI Environments

Agent Simulation to Develop Interactive and User-Centered Conversational Agents

EVALUATION OF A USER-ADAPTED SPOKEN LANGUAGE DIALOGUE SYSTEM - Measuring the Relevance of the Contextual Information Sources

Contact Info

Product

Resources

About