A Corpus Balancing Method for Language Model Construction

Abstract

Abstract. The language model is an important component of any speech recognition system. In this paper, we present a lexical enrichment methodology of corpora focused o n the construction of statistical language models. This methodology co nsiders, on one hand, the identification of the set of poor represented words of a given training corpus, and on the other hand, the enrichment of the given co rpus by the repetitive inclusion of selected text fragments containing these words. The first part of the paper describes the formal details about this methodology; the second part presents some experiments and results that validate our method.

Enhancing Cross-Language Question Answering by Combining Multiple Question Translations

“…This combination is based on the pertinence of the translations to the target document collection. This pertinence, as in the previous method, expresses how a given translation fits in 3 The n-gram model was constructed using the method described in [15].…”

Information Extraction, Search, Interaction and Collaboration on the Web in Mexico

“…In particular, in Mexico there have been some interesting efforts related to the use of the web for the automatic construction of domain-specific ontologies [16], training sets for text classification tasks [6,7], and language models for speech recognition [28]. The following sections give a brief overview of these works.…”

“…The construction of this corpus is not a simple task since written texts do not represent adequately many phenomena of spontaneous speech. In order to alleviate this problem, [28] proposes the use of web documents as data source. This proposal was based on the fact that many people around the world contribute to create the web, and therefore, that most of its documents comprise informal contents and include many everyday as well as non-grammatical expressions used in spoken language.…”

“…In particular, the method presented in [28] faces the problem of enlarging a given small task-specific corpus (called reference corpus). It considers the following main steps:…”

Untitled

“…In addition to Keller and Lapata (this issue) and references therein, Volk (2001) gathers lexical statistics for resolving prepositional phrase attachments, and Villasenor-Pineda et al (2003) "balance" their corpus using Web documents.…”