A Parallelizable Lattice Rescoring Strategy with Neural Language Models

Li, Ké; Povey, Daniel; Khudanpur, Sanjeev

doi:10.48550/arxiv.2103.05081

Cited by 2 publications

(2 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Neural LMs are commonly used in second-pass rescoring [1,2,22,23] or first-pass decoding [3] in ASR systems. While for conventional research-oriented datasets like Switchboard the word-level vocabulary size is several dozens of thousands, for larger systems, especially commercially available systems, the vocabulary size can often go up to several hundred thousand.…”

Section: Related Workmentioning

confidence: 99%

On Sampling-Based Training Criteria for Neural Language Modeling

Gao

Thulke

Gerstenberger

et al. 2021

Preprint

View full text Add to dashboard Cite

As the vocabulary size of modern word-based language models becomes ever larger, many sampling-based training criteria are proposed and investigated. The essence of these sampling methods is that the softmax-related traversal over the entire vocabulary can be simplified, giving speedups compared to the baseline. A problem we notice about the current landscape of such sampling methods is the lack of a systematic comparison and some myths about preferring one over another. In this work, we consider Monte Carlo sampling, importance sampling, a novel method we call compensated partial summation, and noise contrastive estimation. Linking back to the three traditional criteria, namely mean squared error, binary cross-entropy, and crossentropy, we derive the theoretical solutions to the training problems. Contrary to some common belief, we show that all these sampling methods can perform equally well, as long as we correct for the intended class posterior probabilities. Experimental results in language modeling and automatic speech recognition on Switchboard and LibriSpeech support our claim, with all sampling-based methods showing similar perplexities and word error rates while giving the expected speedups.

show abstract

Section: Related Workmentioning

confidence: 99%

On Sampling-Based Training Criteria for Neural Language Modeling

Gao

Thulke

Gerstenberger

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…This flaw of N-best rescoring is mitigated when using rescoring approaches dealing with word lattices. There are approaches such as pruned RNNLM lattice rescoring [16], fast N-best rescoring [17] and parallelizable lattice rescoring [18] which exploit the similarity of hypotheses to optimize computations. Nevertheless, these rescoring approaches are rather slow and computationally expensive as they require multiple LM calls.…”

Section: Introductionmentioning

confidence: 99%

LT-LM: a novel non-autoregressive language model for single-shot lattice rescoring

Mitrofanov¹,

Korenevskaya²,

Podluzhny³

et al. 2021

Preprint

View full text Add to dashboard Cite

Neural network-based language models are commonly used in rescoring approaches to improve the quality of modern automatic speech recognition (ASR) systems. Most of the existing methods are computationally expensive since they use autoregressive language models. We propose a novel rescoring approach, which processes the entire lattice in a single call to the model. The key feature of our rescoring policy is a novel non-autoregressive Lattice Transformer Language Model (LT-LM). This model takes the whole lattice as an input and predicts a new language score for each arc. Additionally, we propose the artificial lattices generation approach to incorporate a large amount of text data in the LT-LM training process. Our single-shot rescoring performs orders of magnitude faster than other rescoring methods in our experiments. It is more than 300 times faster than pruned RNNLM lattice rescoring and N-best rescoring while slightly inferior in terms of WER.

show abstract

A Parallelizable Lattice Rescoring Strategy with Neural Language Models

Cited by 2 publications

References 22 publications

On Sampling-Based Training Criteria for Neural Language Modeling

On Sampling-Based Training Criteria for Neural Language Modeling

LT-LM: a novel non-autoregressive language model for single-shot lattice rescoring

Contact Info

Product

Resources

About