Best-First Beam Search

Meister, Clara; Vieira, Tim; Cotterell, Ryan

doi:10.1162/tacl_a_00346

Cited by 45 publications

(35 citation statements)

References 27 publications

(49 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We hope insights derived from our study stimulate research on tighter integrations between the benefits of cascaded and end-to-end sequence models. Exploiting searchable intermediates through beam search is just the tip of the iceberg for search algorithms, as numerous approximate search techniques like diverse beam search (Vijayakumar et al, 2018) and best-first beam search (Meister et al, 2020) have been recently proposed to improve diversity and approximation of the most-likely sequence. Incorporating differentiable lattice based search (Hannun et al, 2020) can also allow the subsequent sub-net to digest n-best representations.…”

Section: Discussionmentioning

confidence: 99%

Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks

Dalmia¹,

Yan²,

Raunak³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

End-to-end approaches for sequence tasks are becoming increasingly popular. Yet for complex sequence tasks, like speech translation, systems that cascade several models trained on sub-tasks have shown to be superior, suggesting that the compositionality of cascaded systems simplifies learning and enables sophisticated search capabilities. In this work, we present an end-to-end framework that exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks. These hidden intermediates can be improved using beam search to enhance the overall performance and can also incorporate external models at intermediate stages of the network to re-score or adapt towards out-of-domain data. One instance of the proposed framework is a Multi-Decoder model for speech translation that extracts the searchable hidden intermediates from a speech recognition sub-task. The model demonstrates the aforementioned benefits and outperforms the previous state-of-theart by around +6 and +3 BLEU on the two test sets of Fisher-CallHome and by around +3 and +4 BLEU on the English-German and English-French test sets of

show abstract

Section: Discussionmentioning

confidence: 99%

Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks

Dalmia¹,

Yan²,

Raunak³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…To perform bidirectional decoding, we input both of the sentence start tokens to provide more contextual information for the decoder. The specific method, based on that in [34], is illustrated in Algorithm 1. The goal is to search for the best and most likely transcription based on the source sequence.…”

Section: Character Decodingmentioning

confidence: 99%

A Bidirectional Context Embedding Transformer for Automatic Speech Recognition

et al. 2022

View full text Add to dashboard Cite

Transformers have become popular in building end-to-end automatic speech recognition (ASR) systems. However, transformer ASR systems are usually trained to give output sequences in the left-to-right order, disregarding the right-to-left context. Currently, the existing transformer-based ASR systems that employ two decoders for bidirectional decoding are complex in terms of computation and optimization. The existing ASR transformer with a single decoder for bidirectional decoding requires extra methods (such as a self-mask) to resolve the problem of information leakage in the attention mechanism This paper explores different options for the development of a speech transformer that utilizes a single decoder equipped with bidirectional context embedding (BCE) for bidirectional decoding. The decoding direction, which is set up at the input level, enables the model to attend to different directional contexts without extra decoders and also alleviates any information leakage. The effectiveness of this method was verified with a bidirectional beam search method that generates bidirectional output sequences and determines the best hypothesis according to the output score. We achieved a word error rate (WER) of 7.65%/18.97% on the clean/other LibriSpeech test set, outperforming the left-to-right decoding style in our work by 3.17%/3.47%. The results are also close to, or better than, other state-of-the-art end-to-end models.

show abstract

“…Rush et al [37] present a variant of beam search for syntax-and phrase-based MT that comes with guarantees a bound on the possible decoding error and is faster. Meister et al [57] develop a generic reformulation of beam search as agenda-based best-first search. Their implementation is faster than standard implementations and is shown to return the top hypothesis the first time it encounters a complete hypothesis.…”

Section: Algorithmmentioning

confidence: 99%

Decoding Methods in Neural Language Generation: A Survey

2021

View full text Add to dashboard Cite

Neural encoder-decoder models for language generation can be trained to predict words directly from linguistic or non-linguistic inputs. When generating with these so-called end-to-end models, however, the NLG system needs an additional decoding procedure that determines the output sequence, given the infinite search space over potential sequences that could be generated with the given vocabulary. This survey paper provides an overview of the different ways of implementing decoding on top of neural network-based generation models. Research into decoding has become a real trend in the area of neural language generation, and numerous recent papers have shown that the choice of decoding method has a considerable impact on the quality and various linguistic properties of the generation output of a neural NLG system. This survey aims to contribute to a more systematic understanding of decoding methods across different areas of neural NLG. We group the reviewed methods with respect to the broad type of objective that they optimize in the generation of the sequence—likelihood, diversity, and task-specific linguistic constraints or goals—and discuss their respective strengths and weaknesses.

show abstract

Best-First Beam Search

Cited by 45 publications

References 27 publications

Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks

Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks

A Bidirectional Context Embedding Transformer for Automatic Speech Recognition

Decoding Methods in Neural Language Generation: A Survey

Contact Info

Product

Resources

About