Enhancing Context Modeling with a Query-Guided Capsule Network for Document-level Translation

Abstract

Context modeling is essential to generate coherent and consistent translation for Document-level Neural Machine Translations. The widely used method for document-level translation usually compresses the context information into a representation via hierarchical attention networks. However, this method neither considers the relationship between context words nor distinguishes the roles of context words. To address this problem, we propose a query-guided capsule networks to cluster context information into different perspectives from which the target translation may concern. Experiment results show that our method can significantly outperform strong baselines on multiple data sets of different domains.

Dynamic Context Selection for Document-level Neural Machine Translation via Reinforcement Learning

“…Researchers propose various context-aware networks to utilize contextual information to improve the performance of DocNMT models on the translation quality (Jean et al, 2017;Tu et al, 2018;Kuang et al, 2018) or discourse phenomena (Bawden et al, 2018;Voita et al, 2019b,a). However, most methods roughly leverage all context sentences in a fixed size that is tuned on development sets (Wang et al, 2017;Miculicich et al, 2018;Yang et al, 2019;Xu et al, 2020) , or full context in the entire document (Maruf and Haffari, 2018;Tan et al, 2019;Kang and Zong, 2020;Zheng et al, 2020). They ignore the individualized needs for context when translating different source sentences.…”

“…Majority of existing DocNMT models set the context size or scope to be fixed. They utilize all of the previous k context sentences Miculicich et al, 2018;Voita et al, 2019b;Yang et al, 2019;Xu et al, 2020), or the full context in the entire document (Maruf and Haffari, 2018;Tan et al, 2019;Zheng et al, 2020). As a result, the inadequacy or redundancy of contextual information is almost inevitable.…”

Context-Interactive Pre-Training for Document Machine Translation

“…Tu et al (2018) augments translation model with a cache-like memory network that stores recent hidden representations as translation history. Yang et al (2019) introduce a query-guided capsule networks into document-level translation to capture high-level capsules related to the current source sentence. proposes a unified encoder to process the concatenated source information that only attends to the source sentence at the top of encoder blocks.…”

“…Transformer (Vaswani et al, 2017) performs context-agnostic sent-level translation and HAN (Werlen et al, 2018) employs hierarchical attention to capture extra contexts. SAN (Maruf et al, 2019) utilizes top-down attention to selectively focus on relevant sentences and QCN (Yang et al, 2019) uses query-guided capsule networks to capture the related capsulese.…”

Breaking the Corpus Bottleneck for Context-Aware Neural Machine Translation with Cross-Task Pre-training

“…Cache/Memory-based approaches (Tu et al, 2018;Kuang et al, 2018;Maruf and Haffari, 2018;Wang et al, 2017) store word/sentence translation in previous sentences for future sentence translation. Various approaches with an extra context encoders are proposed to model either local context, e.g., previous sentences Wang et al, 2017;Bawden et al, 2018;Voita et al, 2018Voita et al, , 2019bYang et al, 2019;Huo et al, 2020), or entire document (Maruf and Haffari, 2018;Mace and Servan, 2019;Maruf et al, 2019;Tan et al, 2019;Zheng et al, 2020;Kang et al, 2020).…”