1993 benchmark tests for the ARPA spoken language program

Abstract

This paper reports results obtained in benchmark tests conducted within the ARPA Spoken Language program in November and December of 1993. In addition to ARPA contractors, participants included a number of %olunteers", including foreign participants from Canada, France, Germany, and the United Kingdom. The body of the paper is limited to an outline of the structure of the tests and presents highlights and discussion of selected results. Detailed tabulations of reported "official" results, and additional explanatory text appears in the Appendix. 2. WSJ-CSR TESTS 2.1. New Conditions All sites participating in the WSJ-CSR tests were required to submit results for (at least) one of two "Hub" tests. The Hub tests were intended to measure basic speaker-independent performance on either a 64K-word (Hub 1) or 5K-word (Hub 2) read-speech test set, and included required use of either a "standard" 20K trigram (Hub 1) or 5K bigram (Hub 2) grammar, and also required use of standard training sets. These requirements were intended to facilitate meaningful cross-site comparisons. The "Spoke" tests were intended to support a number of different ehaUenges. Spokes 1, 3 and 4 supported problems in various types of adaptation: incremental supervised language model adaptation (Spoke 1), rapid enrollment speaker adaptation for "recognition outliers" (i.e., non-native speakers) (Spoke 3), incremental speaker adaptation (Spoke 4). [There were no participants in what had been planned as Spoke 2.] Spokes 5 through 8 supported problems in noise and channel compensation: unsupervised channel compensation (Spoke 5), "known microphone" adaptation for two different microphones (Spoke 6), unsupervised channel compensation for 2 different environments (Spoke 7), and use of a noise compensation algorithm with a known alternate microphone for data collected in environments when there is competing "calibrated" noise (radio talk shows or music) (Spoke 8). Spoke 9 included spontaneous "dictation-style" speech. Additional details are found in Kubala, et al. [1], on behalf of members of the ARPA Continuous speech recognition Corpus Coordinating Committee (CCCC). 2.2. WSJ-CSR Summary Highlights The design of the "Hub and Spoke" test paradigm, was such that opportunities abounded for informative contrasts (e.g., the use of bigram vs. trigram grammars, the enablement/disablement of supervised vs. unsupervised adaptation strategies, ete). There were nine participating sites in the Hub I tests and five sites participating in the Hub 2 tests, and some sites reported results for more than one system or research team. The lowest word error rate in the Hub 1 baseline condition was achieved by the French CNRS-LIMSI group [2,3]. Application of statistical significance tests indicated that the performance differences between this system and a system

Speaker Adaptation Based on PARAFAC2 of Transformation Matrices for Continuous Speech Recognition

“…These 83 transformation matrices were used to build bases for the EMLLR and the PARAFAC2-based model. For the adaptation test, we used the adaptation data of 8 testing speakers from the WSJ0 corpus, i.e., the November 92 NIST evaluation set [11]. We used 1 to 5 utterances from the adaptation set (an adaptation utterance was about 6 s in length).…”

Survey of the State of the Art in Human Language Technology

“…After working on isolated-word, speaker-dependent systems for many years, since 1992 the community has moved towards very-largevocabulary (20,000 words and more), high-perplexity ( ¡ ¥ ¤ § ¦ ) , speaker-independent, continuous speech recognition. The best system in 1994 achieved an error rate of 7.2% on read sentences drawn from North American business news (Pallett, Fiscus, et al, 1994).…”

“…In the ARPA program, the air travel planning domain has been chosen to support evaluation of spoken language systems (Pallett, 1991;Pallett, 1992;Pallett, Dahlgren, et al, 1992;Pallett, Fisher, et al, 1990;Pallett, Fiscus, et al, 1993;Pallett, Fiscus, et al, 1994;Pallett, Fiscus, et al, 1995). Vocabularies for these systems are usually about 2000 words.…”

Sequence-Based Pronunciation Variation Modeling for Spontaneous ASR Using a Noisy Channel Approach

“…The training set consists of 60 hours of speech and the so-called WSJ test consists of 215 utterances [16]. The WSJ test is a 5k Hub test set.…”