Automatic detection of poor speech recognition at the dialogue level

Abstract

The dialogue strategies used by a spoken dialogue system strongly influence performance and user satisfaction. An ideal system would not use a single fixed strategy, but would adapt to the circumstances at hand. To do so, a system must be able to identify dialogue properties that suggest adaptation. This paper focuses on identifying situations where the speech recognizer is performing poorly. We adopt a machine learning approach to learn rules from a dialogue corpus for identifying these situations. Our results show a significant improvement over the baseline and illustrate that both lower-level acoustic features and higher-level dialogue features can affect the performance of the learning algorithm.

Multi-feature Error Detection in Spoken Dialogue Systems

“…Other studies (Hirschberg, Litman, and Swerts, 1999;Litman, Walker, and Kearns, 1999) reported that the ASR string is highly relevant in predicting recognition errors (which partly corresponds to our CTP task). In the study of (Hirschberg, Litman, and Swerts, 1999) the recognized string was the best performing isolated feature, yielding an error rate of 14.4% (85.6% accuracy).…”

“…Our experiments have indeed shown this discrepancy. (Litman, Walker, and Kearns, 1999) employ the ASR text feature as a setvalued lexical feature in RIPPER where it also turns out to be the most predictive feature in isolation (72% accuracy) for detecting poor speech recognition. It is noteworthy that for our task the ASR string feature is less beneficial.…”

“…Features are selected on grounds of their supposed predictive power towards problems during the interaction between the system and its user. The employed features range from primitive attributes representing entities such as confidence scores output by the automatic speech recognition (ASR) module of the system (Hirschberg, Litman, and Swerts, 1999;Litman, Walker, and Kearns, 1999), lexical output of the ASR module of a SDS (Hirschberg, Litman, and Swerts, 1999;Van den Bosch, Krahmer, and Swerts, 2001), experimental parameters and identification of the underlying ASR grammar (Hirschberg, Litman, and Swerts, 1999;Litman, Walker, and Kearns, 1999), aspects of dialogue efficiency and quality (Litman, Walker, and Kearns, 1999), presence or absence of default assumptions, the amount of slots filled (Krahmer et al, 1999) or the system adaptivity (Hirschberg, Litman, and Swerts, 1999), to highly complicated features, involving a variety of semanticsbased attributes of the user input (Hirschberg, Litman, and Swerts, 1999;Litman, Walker, and Kearns, 1999;Walker, Wright, and Langkilde, 2000), and aspects of syntax in the user answer (Krahmer et al, 1999). As opposed to the primitive at-tributes, the latter types of features cannot be straightforwardly extracted from a SDS, which forms an obstacle for automatic, on-line error detection and recovery.…”

Speech Driven by Artificial Larynx: Potential Advancement Using Synthetic Pitch Contours

“…Strategies for correcting, rejecting, or changing misrecognized hypotheses have been proposed [29,30,31]. Prosodic features such as F0 perturbation, duration, and loudness were shown to significantly characterize failed recognition runs in terms of word-accuracy and conception-accuracy [32].…”

Towards conversational QA

“…To improve dialog performance, much effort has also been put on techniques to automatically detect errors during interaction. It has shown that during human machine dialog, there are sufficient cues for machines to automatically identify error conditions (Levow, 1998;Litman et al, 1999;Hirschberg et al, 2001;Walker et al, 2002). The awareness of erroneous situations can help systems make intelligent decisions about how to best guide human partners through the conversation and accomplish the tasks.…”