Multi-site data collection and evaluation in spoken language understanding

Abstract

The Air Travel Information System (ATIS) domain serves as the common task for DARPA spoken language system research and development. The approaches and results possible in this rapidly growing area are structured by available corpora, annotations of that data, and evaluation methods. Coordination of this crucial infrastructure is the charter of the Multi-Site ATIS Data COllection Working group (MAD-COW). We focus here on selection of training and test data, evaluation of language understanding, and the continuing search for evaluation methods that will correlate well with expected performance of the technology in applications. 1.

The roles of language processing in a spoken language interface.

“…The current evaluation method (4,5) provides an automated evaluation of the correctness of database query responses, presented as prerecorded (speech or transcribed) data in units of "scenarios. "P The data are annotated for their correct reference answers, expressed as a set of minimal and maximal database tuples.…”

“…At Carnegie-Mellon University (CMU), Ward and Young (26) recently reported interesting results based on the tight coupling of a set of recursive transition networks (RTNs) into the recognizer, in conjunction with a bigram language model (26); use of the RTN provided a 20% reduction in both word error and understanding error, compared to the recognizer using the word-class bigram, followed by the RTN for understanding. 5 In experiments at the Massachusetts Institute of Technology (MIT) the TINA language-understanding system was used in a loosely coupled mode to filter N-best output. This produced a very small decrease in word error (0.2%, from 12.7% for N = 1 to 12.5% for N = 5) but a somewhat larger decrease in sentence recognition error (1.7%, from 48.9% for N = 1, to 47.2% for N = 5).h Alternatively, if the language-processing system can provide scores for alternate hypotheses, the hypotheses could be (re)ranked by a weighted combination of recognition and language-understanding score.…”

“…This is a database interface application, where the data were drawn from a nine-city subset of the Official Airline Guide, containing airline, schedule, and ground transportation information.b To support this effort, sites cooperated to collect a training corpus of 14,000 spontaneous utterances (4), and, to date, there have been four formal evaluations in this domain (5)(6)(7)(8)(9). At the start of the Spoken Language Systems program in 1989, an accepted metric had evolved for speech recognition, namely word accuracy (10); however, no comparable metric was available for measuring understanding.…”

Predicting User Satisfaction in Spoken Dialog System Evaluation With Collaborative Filtering

“…After the dialog model issues a proper action as the response, the natural language generator is responsible for translating the representation of the response semantics into text, which is then passed to the Text-to-Speech (TTS) synthesizer to generate audio output. Advances in speech and language technologies have made SDSs an important research area and have brought about systems in a wide variety of application domains, such as bus schedule inquiries [1], flight information [2], stock market information delivery [3], tourist guides [4] and student tutoring [5]. As SDSs are becoming increasingly pervasive, their ultimate goal is to satisfy the users' needs with good performance yielding a good user experience.…”

Experiences of an in-service Wizard-of-Oz data collection for the deployment of a call-routing application

“…One exception from this is the data collection for the original AT&T "How May I Help You" system (Gorin et al 1997;Ammicht et al 1999), which comprised three batches of transactions with live customers, each involving up to 12,000 utterances. Other well-known instances are "Voyager" (Zue et al 1989) and the individual ATIS collections (Hirschman et al 1993) which involved up to a hundred subjects or (again) up to 12,000 utterances.…”