Kneser–Ney Smoothing With a Correcting Transformation for Small Data Sets

Taraba, Peter

doi:10.1109/tasl.2007.900090

Cited by 3 publications

(2 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A number of smoothing algorithms for LM has been investigated. In the literature, a number of smoothing techniques can be seen including Additive smoothing [8], Good-Turing [17], Jelinek-Mercer and Katz smoothing [8], Witten-Bell smoothing [8], Absolute Discounting [13], Kneser Ney [1] and Latent Dirichlet Allocation (LDA) [3]. As per literature survey.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Statistical Approach for Estimating Language Model Reliability with Effective Smoothing Technique

Prajapati¹,

Saha²

2015

IJCA

View full text Add to dashboard Cite

Language Model smoothing is an imperative technology which deals with unseen test data by re-evaluating some zeroprobability n-grams and assign them bare minimum non-zero values. There is an assortment of smoothing techniques employed to trim down tiny amount of probability from the probable grams and share out to zero probable grams within a Language Model. Kneser Ney and Latent Dirichlet Allocation algorithm are two probable techniques used for proficient smoothing. In this paper, a scheme is proposed for effective smoothing by combining Kneser Ney and Latent Dirichlet Allocation approaches. Moreover, another scheme is proposed to measure the reliability of a Language Model and determine the association between entropy and perplexity. These schemes are demonstrated by appropriate examples.

show abstract

Section: Introductionmentioning

confidence: 99%

“…Kneser Ney smoothing technique has been evolved from Absolute Discounting technique [13] which assimilates the information of both higher order and lower order of the gram to assign the probability. It relies on the context of the gram to assign the probability.…”

Section: Introductionmentioning

confidence: 99%