CMU’s IWSLT 2022 Dialect Speech Translation System

Yan, Brian; Fernandes, Patrick; Dalmia, Siddharth; Shi, Juntai; Peng, Yifan; Berrebbi, Dan; Wang, Xinyi; Neubig, Graham; Watanabe, Shinji

doi:10.18653/v1/2022.iwslt-1.27

Cited by 8 publications

(11 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lattice MBR (Tromble et al, 2008) N-best list N-best list BLEU translation lattice Coarse-to-fine MBR (Eikema and Aziz, 2022) ancestral sampling filter(sample) BEER single model Wiher et al (2022) ancestral sampling evidence + more decodings BEER single model MBR-DC (Yan et al, 2023) temperature sampling 1 temperature sampling 1 BLEURT single model Ours ( § 3.3) ancestral sampling temperature sampling BERTScore single model Ours ( § 3.4) ancestral sampling temperature sampling BERTScore length-corrected scores Freitag et al (2023) epsilon sampling BLEURT single model Crowd sampling 2 (Suzgun et al, 2023) temperature sampling neural score metric single model MBR-Exec (Shi et al, 2022) temperature sampling execution match single model…”

Section: Methodsmentioning

confidence: 99%

“…The gain G (alternatively, error L) may be an arbitrary function Y e × Y h → R. Early work focused on simple, token-level metrics like word error rate and BLEU (Kumar and Byrne, 2004;Ehling et al, 2007), but more recent work has explored the use of neural metrics (Amrhein and Sennrich, 2022;Freitag et al, 2022), as well as executing outputs in code generation (Shi et al, 2022;Li et al, 2022).…”

Section: What Metric Do We Want To Maximize?mentioning

confidence: 99%

See 1 more Smart Citation

It’s MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk

Bertsch,

Xie,

Neubig

et al. 2023

Proceedings of the Big Picture Workshop

View full text Add to dashboard Cite

Minimum Bayes Risk (MBR) decoding is a method for choosing the outputs of a machine learning system based not on the output with the highest probability, but the output with the lowest risk (expected error) among multiple candidates. It is a simple but powerful method: for an additional cost at inference time, MBR provides reliable several-point improvements across metrics for a wide variety of tasks without any additional data or training. Despite this, MBR is not frequently applied in NLP works, and knowledge of the method itself is limited. We first provide an introduction to the method and the recent literature. We show that several recent methods that do not reference MBR can be written as special cases of MBR; this reformulation provides additional theoretical justification for the performance of these methods, explaining some results that were previously only empirical. We provide theoretical and empirical results about the effectiveness of various MBR variants and make concrete recommendations for the application of MBR in NLP models, including future directions in this area.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: What Metric Do We Want To Maximize?mentioning

confidence: 99%

It’s MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk

Bertsch,

Xie,

Neubig

et al. 2023

Proceedings of the Big Picture Workshop

View full text Add to dashboard Cite

show abstract

“…MBR has since been extensively in used in submissions to various machine and speech translation shared tasks with good results (Nowakowski et al, 2022;Jon et al, 2022;Yan et al, 2022), showcasing its potential to improve translation quality.…”

Section: Related Workmentioning

confidence: 99%

Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation

Freitag,

Ghorbani,

Fernandes

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Recent advances in machine translation (MT) have shown that Minimum Bayes Risk (MBR) decoding can be a powerful alternative to beam search decoding, especially when combined with neural-based utility functions. However, the performance of MBR decoding depends heavily on how and how many candidates are sampled from the model. In this paper, we explore how different sampling approaches for generating candidate lists for MBR decoding affect performance. We evaluate popular sampling approaches, such as ancestral, nucleus, and top-k sampling. Based on our insights into their limitations, we experiment with the recently proposed epsilon-sampling (Hewitt et al., 2022) approach, which prunes away all tokens with a probability smaller than epsilon, ensuring that each token in a sample receives a fair probability mass. Through extensive human evaluations, we demonstrate that MBR decoding based on epsilon-sampling significantly outperforms not only beam search decoding, but also MBR decoding with all other tested sampling methods across four language pairs.

show abstract

“…Speculative beam search is employed at inference time. HW-TSC (Wang et al, 2022) entered all tracks, i.e. speech-to-text and text-to-text for English-German, English-Japanese and English-Mandarin.…”

Section: Submissionsmentioning

confidence: 99%

“…CMU (Yan et al, 2022) focuses on the Multi-Decoder architecture (Dalmia et al, 2021) implemented in ESPnet, which is an end-to-end ST model that decomposes into ASR and MT subnets while maintaining differentiability. Intuitively, hidden states found by beam search from the ASR decoder are fed as input to the ST encoder.…”

Section: Submissionsmentioning

confidence: 99%

Findings of the IWSLT 2022 Evaluation Campaign

Anastasopoulos¹,

Barrault²,

Bentivogli³

et al. 2022

Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)

Self Cite

View full text Add to dashboard Cite

The evaluation campaign of the 19th International Conference on Spoken Language Translation featured eight shared tasks: (i) Simultaneous speech translation, (ii) Offline speech translation, (iii) Speech to speech translation, (iv) Low-resource speech translation, (v) Multilingual speech translation, (vi) Dialect speech translation, (vii) Formality control for speech translation, (viii) Isometric speech translation. A total of 27 teams participated in at least one of the shared tasks. This paper details, for each shared task, the purpose of the task, the data that were released, the evaluation metrics that were applied, the submissions that were received and the results that were achieved.

show abstract

CMU’s IWSLT 2022 Dialect Speech Translation System

Cited by 8 publications

References 0 publications

It’s MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk

It’s MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk

Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation

Findings of the IWSLT 2022 Evaluation Campaign

Contact Info

Product

Resources

About