Linguistically-augmented perplexity-based data selection for language models

Abstract

This paper explores the use of linguistic information for the selection of data to train language models. We depart from the stateof-the-art method in perplexity-based data selection and extend it in order to use word-level linguistic units (i.e. lemmas, named entity categories and part-of-speech tags) instead of surface forms. We then present two methods that combine the different types of linguistic knowledge as well as the surface forms (1, naïve selection of the top ranked sentences selected by each method; 2, linear interpolation of the datasets selected by the different methods). The paper presents detailed results and analysis for four languages with different levels of morphologic complexity (English, Spanish, Czech and Chinese). The interpolation-based combination outperforms the purely statistical baseline in all the scenarios, resulting in language models with lower perplexity. In relative terms the improvements are similar regardless of the language, with perplexity reductions achieved in the range 7.72-13.02%. In absolute terms the reduction is higher for languages with high type-token ratio (Chinese, 202.16) or rich morphology (Czech, 81.53) and lower for the remaining languages, Spanish (55.2) and English (34.43 on the English side of the same parallel dataset as for Czech and 61.90 on the same parallel dataset as for Spanish).

Efficient Search Mechanism from Large Scale Corpora for Domain-Specific Language Modeling in Speech Recognition

“…LDA technique has been widely explored to form unsupervised adapted language model [17] and topic-specific language models for inflectional languages [18]. For many languages the linguistic word level approach [19], syntactico-statistical approach [19] and statistic phrase level approach [20] has been used to build an adapted language model for improving the speech recognition rate. Document retrieval from web content [21, 22.…”

Ameliorated language modelling for lecture speech recognition of Indian English

“…Many linguistically rich languages have come up with the word-level linguistic approach for the generation of a better LM. Toral et al [15] have used word-level linguistic units such as lemmas, Named Entity Recognition (NER) and Parts of Speech (POS) tags. In this paper [15], two kinds of LMs are created: domain-specific corpus and random subset of general corpus, which is of same size as the domainspecific corpus.…”

Data Selection for IT Texts using Paragraph Vector

“…In Mansour et al (2011), the cross-entropy score is used for language model filtering together with a translation model score that estimates the likelihood that a source and a target sentence are a translation of each other. Toral et al (2015) introduced linguistic information such as lemmas, named entities and part-of-speech tags into the preprocessing of the data and then ranked the sentences by perplexity.…”