Improved language modelling through better language model evaluation measures

Clarkson, Philip; Robinson, Tony

doi:10.1006/csla.2000.0156

Cited by 11 publications

(13 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…It is supposed that the "best" models get the "lowest" Word Error Rates (WER) in the CSR system, but there are many contra examples in literature (Rosenfeld, 2000). The ability of the test set perplexity to predict the real behavior of a smoothing technique when the smoothed LM is working into a CSR system could be questioned (Clarkson and Robinson, 1999) since it does not take into account the relationship with acoustic models. Several attempts have been made to devise metrics that are better correlated with Word Error Rates than perplexity (Clarkson and Robinson, 1999;Bimbot et al, 2001), but for now perplexity remains the main metric for practical language model construction (Rosenfeld, 2000).…”

Section: Introductionmentioning

confidence: 99%

“…The ability of the test set perplexity to predict the real behavior of a smoothing technique when the smoothed LM is working into a CSR system could be questioned (Clarkson and Robinson, 1999) since it does not take into account the relationship with acoustic models. Several attempts have been made to devise metrics that are better correlated with Word Error Rates than perplexity (Clarkson and Robinson, 1999;Bimbot et al, 2001), but for now perplexity remains the main metric for practical language model construction (Rosenfeld, 2000). In fact, the quality of the model must be ultimately measured by its effect on the specific task for which it was designed, namely by its effect on the system error rate.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Scaling Smoothed Language Models

Varona¹,

Torres²

2005

Int J Speech Technol

View full text Add to dashboard Cite

In Continuous Speech Recognition (CSR) systems a Language Model (LM) is required to represent the syntactic constraints of the language. Then a smoothing technique needs to be applied to avoid null LM probabilities. Each smoothing technique leads to a different LM probability distribution. Test set perplexity is usually used to evaluate smoothing techniques but the relationship with acoustic models is not taken into account. In fact, it is well-known that to obtain optimum CSR performances a scaling exponential parameter must be applied over LMs in the Bayes' rule. This scaling factor implies a new redistribution of smoothed LM probabilities. The shape of the final probability distribution is due to both the smoothing technique used when designing the language model and the scaling factor required to get the optimum system performance when integrating the LM into the CSR system. The main object of this work is to study the relationship between the two factors, which result in dependent effects. Experimental evaluation is carried out over two Spanish speech application tasks. Classical smoothing techniques representing very different degrees of smoothing are compared. A new proposal, Delimited discounting, is also considered. The results of the experiments showed a strong dependence between the amount of smoothing given by the smoothing technique and the way that the LM probabilities need to be scaled to get the best system performance, which is perplexity independent in many cases. This relationship is not independent of the task and available training data.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Scaling Smoothed Language Models

Varona¹,

Torres²

2005

Int J Speech Technol

View full text Add to dashboard Cite

show abstract

“…The ability of the test set perplexity to predict the real behavior of a smoothing technique when working in a CSR system could be questioned because it does not take into account the relationship with acoustic models. Several attempts have been made to devise metrics that are better correlated with the application error rate than perplexity [4]. But for now perplexity remains the main metric for practical language model construction [3].…”

Section: Introductionmentioning

confidence: 99%

Integrating High and Low Smoothed LMs in a CSR System

Varona

Torres

2003

Lecture Notes in Computer Science

View full text Add to dashboard Cite

In Continuous Speech Recognition (CSR) systems, acoustic and Language Models (LM) must be integrated. To get optimum CSR performances, it is well-known that heuristic factors must be optimised. Due to its great effect on final CSR performances, the exponential scaling factor applied to LM probabilities is the most important. LM probabilities are obtained after applying a smoothing technique. The use of the scaling factor implies a redistribution of the smoothed LM probabilities, i.e., a new smoothing is obtained. In this work, the relationship between the amount of smoothing of LMs and the new smoothing achieved by the scaling factor is studied. High and low smoothed LMs, using wellknown discounting techniques, were integrated into the CSR system. The experimental evaluation was carried out on two Spanish speech application tasks with very different levels of difficulty. The strong relationship observed between the two redistributions of the LM probabilities was independent of the task. When the adequate value of the scaling factor was applied, not very different optimum CSR performances were obtained in spite of the great differences between perplexity values.

show abstract

“…La correlación entre PPL y WER/BLEU es un tema que ha sido investigado por muchos autores [Iyer et al, 1997;Chen et al, 1998;Clarkson & Robinson, 1999;Ito et al, 1999;…”

Section: Correlación Entre Ppl Y Werunclassified

“…The correlation between PPL and the final system error (measured as WER, BLEU, or TER) is very task dependent and is not totally clear. Some authors made studies of the relationship between PPL and WER [Iyer et al, 1997;Chen et al, 1998;Clarkson & Robinson, 1999;Ito et al, 1999;Printz & Olsen, 2002;Klakow & Peters, 2002], concluding that the correlation is hidden by the test set characteristics, nevertheless the correlation exists. As an illustrative example, Figure 2.1 on page 30 shows the evolution of WER vs PPL on the IAM task [Marti & Bunke, 1999] of HTR and the French Media task [Bonneau-Maynard et al, 2005] of ASR.…”

Section: D21 Statistical Language Modelsmentioning

confidence: 99%

Aportaciones al modelado conexionista de lenguaje y su aplicación al reconocimiento de secuencias y traducción automática

Martínez¹,

Julián²

View full text Add to dashboard Cite

A Josepa por apoyarme en estos últimos años de escritura. A mis padres, Natividad y Crescencio, y hermanos, José y Nati, por creer en mí. AGRADECIMIENTOSQuiero aprovechar este espacio para agradecer el apoyo, críticas, y buenas intenciones que he encontrado en la gente en el camino de obtener el título de Doctor. Doy las gracias a mis amigos de Aldaia, que he tenido muy olvidados en estos últimos años, pero que siguen confiando en mí. A Germán, Jorge y Jesús compañeros de andadura durante la carrera, y compañeros en el doctorado, gracias por las discusiones, comentarios, curiosidades y demás cosas que he podido aprender con vosotros. A Lucena, Ximo, Carlos, Julien, Fidel, Antonio, Pablo por los años de música y conciertos. A Noelia por acompañarme durante un tiempo en la vida solapado con esta tesis. A Salva por estos años haciendo guía y ayudando en todo lo que ha podido siempre, aportando un punto de vista original, implicados ambos en la programación del software aquí utilizado. A María José por la dirección desde mi proyecto final de carrera, hasta el final con las sucesivas correcciones que esta tesis ha necesitado para ser lo que es. A Jon, Emilio y Encarna por su apoyo en diferentes ocasiones para encontrar trabajo. A la gente que ha pasado por el laboratorio L302, Sergio, David, Joan, Marcos, Lucía, Ximo, por su amistad. A la gente del departamento de Ciencias Físicas, Matemáticas y de la Computación de la Universidad CEU Cardenal Herrera, donde actualmente estoy trabajando, por haberme dado la oportunidad de poder poner un pie en el mundo de la enseñanza universitaria, a Javi, Nuria Rosillo, Juan, Pablo, Rafa Pérez, Rafa Ramírez, Nuria Llull, Manolo, Luís, Victor, Jordi, Antonio, Fernando, que juntos nos esforzamos por dar lo mejor en clase, y que me han ofrecido toda su confianza para que termine esta tesis. A los alumnos Fran, Joan, Vicent y Jose Manuel por implicarse en la competición del SWERC, que tanto disfruto. A la gente del DSIC, así como a gente del ITI, que he ido conociendo estos años. A Renato de Mori por su valiosa aportación a este trabajo. A la gente del LIUM en Le Mans, Patrik, Loïc, Kashif, Benoit, por ofrecerme su amistad durante mi estancia veraniega del 2010 en su universidad, y por supuesto a Holger Schwenk por acogerme y apoyar mis ideas. A Maxim por introducirme en el mundo de la traducción automática, y Josep María por recibirme en aquel verano del 2010. Al antiguo grupo IAM de la Universidad de Bern, Volkmar Frinken, Andreas Fischer y Horst Bunke, por los trabajos colaborativos que han dado una mayor profundidad a los resultados presentados en esta tesis. A todos aquellos de los que puede que me olvide que han repercutido de alguna forma en este trabajo, y que me han apoyado para llegar hasta aquí. A todos vosotros, ¡Gracias! V RESUMEN El procesamiento del lenguaje natural es un área de la inteligencia artificial, en particular, del reconocimiento de formas. Es un campo multidisciplinar que estudia el lenguaje humano, tanto oral como escrito. Se ocupa de la formulación e invest...

show abstract

Improved language modelling through better language model evaluation measures

Cited by 11 publications

References 5 publications

Scaling Smoothed Language Models

Scaling Smoothed Language Models

Integrating High and Low Smoothed LMs in a CSR System

Aportaciones al modelado conexionista de lenguaje y su aplicación al reconocimiento de secuencias y traducción automática

Contact Info

Product

Resources

About