Locality-Sensitive Hashing for Long Context Neural Machine Translation

Petrick, Frithjof; Rosendahl, Jan; Herold, Christian; Ney, Hermann

doi:10.18653/v1/2022.iwslt-1.4

Cited by 2 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In general, the papers discussed so far have looked at evaluating or improving performance of LLMs as translation systems. Petrick et al (2023) explore ways of fusing document-level language models with NMT systems, including LLMs. This provides an example of ways we might see knowledge from LLMs incorporated into dedicated MT systems, an avenue that could be explored in parallel to that of treating LLMs as MT systems.…”

Section: Llms For Translationmentioning

confidence: 99%

A survey of context in neural machine translation and its evaluation

Castilho,

Knowles

2024

Nat. lang. processing

View full text Add to dashboard Cite

The question of context in neural machine translation often focuses on topics related to document-level translation or intersentential context. However, there is a wide range of other aspects that can be considered under the umbrella of context. In this work, we survey ways that researchers have incorporated context into neural machine translation systems and the evaluation thereof. This includes building translation systems that operate at the paragraph level or the document level or ones that translate at the sentence level but incorporate information from other sentences. We also consider how issues like terminology consistency, anaphora, and world knowledge or external information can be considered as types of context relevant to the task of machine translation and its evaluation. Closely tied to these topics is the question of how to best evaluate machine translation output in a way that is sensitive to the contexts in which it appears. To this end, we discuss work on incorporating context into both human and automatic evaluations of machine translation quality. Furthermore, we also discuss recent experiments in the field as they relate to the use of large language models in translation and evaluation. We conclude with a view of the future of machine translation, where we expect to see issues of context continue to come to the forefront.

show abstract

Section: Llms For Translationmentioning

confidence: 99%

A survey of context in neural machine translation and its evaluation

Castilho,

Knowles

2024

Nat. lang. processing

View full text Add to dashboard Cite

show abstract

“…• LSH-trans (Petrick et al, 2022) 1 is based on Reformer and uses locality sensitive hashing to obtain a cluster of tokens to be attended to each other within it.…”

Section: Comparison Workmentioning

confidence: 99%

“…On the other hand, the studies that target on efficient sequence-to-sequence generation only verify their methods on normal sentence-level translation benchmarks like WMT EN-DE test sets (Peng et al, 2021;Petrick et al, 2022;Ma et al, 2021). In our preliminary experiments, we find that almost all the work severely drops in BLEU when dealing with real document translation tasks.…”

Section: Introductionmentioning

confidence: 99%

Only 5% Attention Is All You Need: Efficient Long-range Document-level Neural Machine Translation

Liu,

Sun,

Cheng

et al. 2023

Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacifi

View full text Add to dashboard Cite

Document-level Neural Machine Translation (DocNMT) has been proven crucial for handling discourse phenomena by introducing document-level context information. One of the most important directions is to input the whole document directly to the standard Transformer model. In this case, efficiency becomes a critical concern due to the quadratic complexity of the attention module. Existing studies either focus on the encoder part, which cannot be deployed on sequence-to-sequence generation tasks, e.g., Machine Translation (MT), or suffer from a significant performance drop. In this work, we keep the translation performance while gaining 20% speed up by introducing extra selection layer based on lightweight attention that selects a small portion of tokens to be attended. It takes advantage of the original attention to ensure performance and dimension reduction to accelerate inference. Experimental results show that our method could achieve up to 95% sparsity (only 5% tokens attended) approximately, and save 93% computation cost on the attention module compared with the original Transformer, while maintaining the performance. * * Work was done while Z. Liu was an intern at ByteDance.

show abstract

Locality-Sensitive Hashing for Long Context Neural Machine Translation

Cited by 2 publications

References 0 publications

A survey of context in neural machine translation and its evaluation

A survey of context in neural machine translation and its evaluation

Only 5% Attention Is All You Need: Efficient Long-range Document-level Neural Machine Translation

Contact Info

Product

Resources

About