Issues in the evaluation of spoken dialogue systems using objective and subjective measures

Larsen, Lars Bo

doi:10.1109/asru.2003.1318442

Cited by 19 publications

(14 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…confirming a purchase with a yes/no question), however this is not always possible and user responses can be noisy [8] which results in slower learning.…”

Section: Introductionmentioning

confidence: 96%

Multi-domain dialogue success classifiers for policy training

Vandyke

Gašić

et al. 2015

2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)

View full text Add to dashboard Cite

We propose a method for constructing dialogue success classifiers that are capable of making accurate predictions in domains unseen during training. Pooling and adaptation are also investigated for constructing multi-domain models when data is available in the new domain. This is achieved by reformulating the features input to the recurrent neural network models introduced in [1]. Importantly, on our task of main interest, this enables policy training in a new domain without the dialogue success classifier (which forms the reinforcement learning reward function) ever having seen data from that domain before. This occurs whilst incurring only a small reduction in performance relative to developing and using an in-domain dialogue success classifier. Finally, given the motivation with these dialogue success classifiers is to enable policy training with real users, we demonstrate that these initial policy training results obtained with a simulated user carry over to learning from paid human users.Index Terms-statistical spoken dialogue systems, dialogue success, multi-domain, policy training

show abstract

“…confirming a purchase with a yes/no question), however this is not always possible and user responses can be noisy [8] which results in slower learning.…”

Section: Introductionmentioning

confidence: 96%

Multi-domain dialogue success classifiers for policy training

Vandyke

Gašić

et al. 2015

2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)

View full text Add to dashboard Cite

show abstract

“…Recently, PARADISE was the subject of several investigations, among others (Aguilera et al, 2004;Larsen, 2003b;Paek, 2001;Whittaker, Terveen, & Nardi, 2000). The main limitation found was that tasks have to be clearly defined so that they can be described by an AVM.…”

Section: Discussionmentioning

confidence: 95%

Adaptation of an automotive dialogue system to users’ expertise and evaluation of the system

Hassel

Hagen

2006

Lang Resources & Evaluation

View full text Add to dashboard Cite

Spoken dialogue systems (SDSs) can be used to operate devices, e.g. in the automotive environment. People using these systems usually have different levels of experience. However, most systems do not take this into account. In this paper, we present a method to build a dialogue system in an automotive environment that automatically adapts to the user's experience with the system. We implemented the adaptation in a prototype and carried out exhaustive tests. Our usability tests show that adaptation increases both user performance and user satisfaction. We describe the tests that were performed, and the methods used to assess the test results. One of these methods is a modification of PARADISE, a framework for evaluating the performance of SDSs [Walker MA, Litman DJ, Kamm CA, Abella A (Comput Speech Lang 12(3):317-347, 1998)]. We discuss its drawbacks for the evaluation of SDSs like ours, the modifications we have carried out, and the test results.

show abstract

“…These aspects can be used as the basis for usability evaluation strategies. Many frameworks and methodologies have been developed and used for evaluation of spoken dialogue systems in recent works [8,9,13,15,17,18,21,24,30].…”

Section: 12mentioning

confidence: 99%

Spoken Dialogue Interfaces: Integrating Usability

Spiliotopoulos

Stavropoulou

2009

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Usability is a fundamental requirement for natural language interfaces. Usability evaluation reflects the impact of the interface and the acceptance from the users. This work examines the potential of usability evaluation in terms of issues and methodologies for spoken dialogue interfaces along with the appropriate designer-needs analysis. It unfolds the perspective to the usability integration in the spoken language interface design lifecycle and provides a framework description for creating and testing usable content and applications for conversational interfaces. Main concerns include the problem identification of design issues for usability design and evaluation, the use of customer experience for the design of voice interfaces and dialogue, and the problems that arise from real-life deployment. Moreover it presents a real-life paradigm of a hands-on approach for applying usability methodologies in a spoken dialogue application environment to compare against a DTMF approach. Finally, the scope and interpretation of results from both the designer and the user standpoint of usability evaluation are discussed.

show abstract

Issues in the evaluation of spoken dialogue systems using objective and subjective measures

Cited by 19 publications

References 2 publications

Multi-domain dialogue success classifiers for policy training

Multi-domain dialogue success classifiers for policy training

Adaptation of an automotive dialogue system to users’ expertise and evaluation of the system

Spoken Dialogue Interfaces: Integrating Usability

Contact Info

Product

Resources

About