The hub and spoke paradigm for CSR evaluation

Abstract

In this paper, we introduce the new paradigm used in the most recent ARPA-sponsored Continuous Speech Recognition (CSR) evaluation and then discuss the important features of the test design.The 1993 CSR evaluation was organized in a novel fashion in an attempt to accomodate research over a broad variety of important problems in CSR while maintaining a clear program-wide research focus. Furthermore, each test component in the evaluation was designed as an experiment to extract as much information as possible from the results.The evaluation was centered around a large vocabulary speakerindependent (SI) baseline test, which was required of every participating site. This test was dubbed the 'Hub' since it was common to all sites and formed the basis for controlled inter-system com- parisons.The Hub test was augmented with a variety of problem-specific optional tests designed to explore a variety of important problems in CSR, mostly involving some kind of mismatch between the training and test conditions. These tests were known as the 'Spokes' since they all could be informatively compared to the Hub, but were otherwise independent.In the first trial of this evaluation paradigm in November, 1993, 11 research groups participated, yielding a rich array of comparative and contrastive results, all calibrated to the current state of the art in large vocabulary CSR.

Exploiting latent semantic information in statistical language modeling

“…The general domain considered was business news, as reflected in the WSJ portion of the NAB corpus. This was convenient for comparison purposes since conventional -gram language models are readily available, trained on exactly the same data [46].…”

“…This is the case, for example, with the ARPA North American Business (NAB) News corpus [46]. Once this is done, the LSA paradigm can be used for word and document clustering [12], [34], [36], as well as for language modeling [3], [6], [24].…”

“…Let , , be some underlying vocabulary and a training text corpus, comprising articles (documents) relevant to some domain of interest (like business news, e.g., in the case of the NAB corpus [46]). Typically, and may be on the order of 10 000 and 100 000, respectively; might comprise a hundred million words or so.…”

Online adaptation of hidden Markov models using incremental estimation algorithms

“…We evaluated batch and online adaptation on the "spoke 3" task of the large-vocabulary WSJ corpus [20], [22]. The goal of this task is to improve recognition performance for nonnative speakers of American English.…”

Speech recognition evaluation: a review of the U.S. CSR and LVCSR programmes

“…Secondly, there was strong interest in investigating the effects of differences in test material and conditions such as non-native speech, multiple microphones, background noise, etc. Since no single participant could evaluate on all possible contrasts, a new ''Hub and Spoke Paradigm'' was developed (Kubala et al, 1994).…”