Metrics and Evaluation of Spoken Dialogue Systems

Hastie, Helen Wright

doi:10.1007/978-1-4614-4803-7_7

Cited by 19 publications

(16 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Dialogue Dialogue evaluation is relatively well understood in goal-oriented tasks, where automated approaches can be coded by measuring task completion El Asri et al, 2017;Hastie, 2012;Henderson et al, 2014;Wen et al, 2017). Task success combined with dialogue cost can be linked to human judgments like user satisfaction via the PARADISE framework (Walker et al, 1997).…”

Section: Related Workmentioning

confidence: 99%

What makes a good conversation? How controllable attributes affect human judgments

See¹,

Roller²,

Kiela³

et al. 2019

Proceedings of the 2019 Conference of the North

206

281

View full text Add to dashboard Cite

A good conversation requires balance -between simplicity and detail; staying on topic and changing it; asking questions and answering them. Although dialogue agents are commonly evaluated via human judgments of overall quality, the relationship between quality and these individual factors is less well-studied. In this work, we examine two controllable neural text generation methods, conditional training and weighted decoding, in order to control four important attributes for chitchat dialogue: repetition, specificity, response-relatedness and question-asking. We conduct a large-scale human evaluation to measure the effect of these control parameters on multi-turn interactive conversations on the PersonaChat task. We provide a detailed analysis of their relationship to high-level aspects of conversation, and show that by controlling combinations of these variables our models obtain clear improvements in human quality judgments.

show abstract

Section: Related Workmentioning

confidence: 99%

What makes a good conversation? How controllable attributes affect human judgments

See¹,

Roller²,

Kiela³

et al. 2019

Proceedings of the 2019 Conference of the North

206

281

View full text Add to dashboard Cite

show abstract

“…Only the performance of Converness is presented in this paper, that is, the ability to recognise the discussion topic and couple profile information. Other aspects of the dialogue‐based agent (Hastie, ), that is, DM, multimodal communication analysis, language generation, are out of the scope of this paper (external modules have been used, as described in the next section), and they are not part of the Converness evaluation.…”

Section: Experimental Evaluationmentioning

confidence: 99%

Converness: Ontology‐driven conversational awareness and context understanding in multimodal dialogue systems

et al. 2019

View full text Add to dashboard Cite

Dialogue‐based systems often consist of several components, such as communication analysis, dialogue management, domain reasoning, and language generation. In this paper, we present Converness, an ontology‐driven, rule‐based framework to facilitate domain reasoning for conversational awareness in multimodal dialogue‐based agents. Converness uses Web Ontology Language 2 (OWL 2) ontologies to capture and combine the conversational modalities of the domain, for example, deictic gestures and spoken utterances, fuelling conversational topic understanding, and interpretation using description logics and rules. At the same time, defeasible rules are used to couple domain and user‐centred knowledge to further assist the interaction with end users, facilitating advanced conflict resolution and personalised context disambiguation. We illustrate the capabilities of the framework through its integration into a multimodal dialogue‐based agent that serves as an intelligent interface between users (elderly, caregivers, and health experts) and an ambient assistive living platform in real home settings.

show abstract

“…Design principles should provide guidelines to improve and maintain the quality of CAs. How to measure the quality of task-orientated dialog systems, as discussed above, has been much researched (see [19] for an overview). Evaluation of non-task oriented social dialog systems, on the other hand, is a new emerging challenge, as there is no clear measure for task success and evaluating whether rapport has been established is far from clear-cut [9,39].…”

Section: Design Principles For Conversational Agentsmentioning

confidence: 99%

Exploring Interaction with Remote Autonomous Systems using Conversational Agents

Robb

Lopes

Padilla

et al. 2019

Proceedings of the 2019 on Designing Interactive Systems Conference

Self Cite

View full text Add to dashboard Cite

Autonomous vehicles and robots are increasingly being deployed to remote, dangerous environments in the energy sector, search and rescue and the military. As a result, there is a need for humans to interact with these robots to monitor their tasks, such as inspecting and repairing offshore wind-turbines. Conversational Agents can improve situation awareness and transparency, while being a hands-free medium to communicate key information quickly and succinctly. As part of our user-centered design of such systems, we conducted an indepth immersive qualitative study of twelve marine research scientists and engineers, interacting with a prototype Conversational Agent. Our results expose insights into the appropriate content and style for the natural language interaction and, from this study, we derive nine design recommendations to inform future Conversational Agent design for remote autonomous systems.

show abstract

Metrics and Evaluation of Spoken Dialogue Systems

Cited by 19 publications

References 34 publications

What makes a good conversation? How controllable attributes affect human judgments

What makes a good conversation? How controllable attributes affect human judgments

Converness: Ontology‐driven conversational awareness and context understanding in multimodal dialogue systems

Exploring Interaction with Remote Autonomous Systems using Conversational Agents

Contact Info

Product

Resources

About