Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite

Jwalapuram, Prathyusha; Joty, Shafiq; Temnikova, Irina; Nakov, Preslav

doi:10.18653/v1/d19-1294

Cited by 15 publications

(17 citation statements)

References 31 publications

(26 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although neural machine translation (NMT) has achieved great progress in recent years (Cho et al, 2014;Bahdanau et al, 2015;Luong et al, 2015;Vaswani et al, 2017), when fed an entire document, standard NMT systems translate sentences in isolation without considering the cross-sentence dependencies. Consequently, document-level neural machine translation (DocNMT) methods are proposed to utilize source-side or target-side intersentence contextual information to improve translation quality over sentences in a document (Jean et al, 2017;Wang et al, 2017;Tiedemann and Scherrer, 2017;Tu et al, 2018;Kuang et al, 2018;Junczys-Dowmunt, 2019;Ma et al, 2020 More recently, researchers of DocNMT mainly focus on exploring various attention-based networks to leverage the cross-sentence context efficiently, and evaluate the special discourse phenomena (Bawden et al, 2018;Müller et al, 2018;Voita et al, 2019b;Jwalapuram et al, 2019). However, there is still an issue that has received less attention: which context sentences should be used when translating a source sentence?…”

Section: Introductionmentioning

confidence: 99%

Dynamic Context Selection for Document-level Neural Machine Translation via Reinforcement Learning

Kang¹,

Zhao²,

Zhang³

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Document-level neural machine translation has yielded attractive improvements. However, majority of existing methods roughly use all context sentences in a fixed scope. They neglect the fact that different source sentences need different sizes of context. To address this problem, we propose an effective approach to select dynamic context so that the document-level translation model can utilize the more useful selected context sentences to produce better translations. Specifically, we introduce a selection module that is independent of the translation module to score each candidate context sentence. Then, we propose two strategies to explicitly select a variable number of context sentences and feed them into the translation module. We train the two modules end-to-end via reinforcement learning. A novel reward is proposed to encourage the selection and utilization of dynamic context sentences. Experiments demonstrate that our approach can select adaptive context sentences for different source sentences, and significantly improves the performance of document-level translation methods.

show abstract

Section: Introductionmentioning

confidence: 99%

Dynamic Context Selection for Document-level Neural Machine Translation via Reinforcement Learning

Kang¹,

Zhao²,

Zhang³

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

show abstract

“…To eliminate word alignment errors, we compare this overlap over the set of dictionarymatched target pronouns, in contrast to the set of target words aligned to a given source pronoun as done by AutoPRF and APT. two measures which rely on computing pronoun overlap between the target and reference translation, we employ an ELMo-based (Peters et al, 2018) evaluation framework that distinguishes between a good and a bad translation via pairwise ranking (Jwalapuram et al, 2019). We use the CRC setting of this metric which considers the same reference context (one previous and one next sentence) for both reference and system translations.…”

Section: Discussionmentioning

confidence: 99%

“…. Common Reference Context (CRC)(Jwalapuram et al, 2019). In addition to the previous 9 https://github.com/idiap/APT…”

mentioning

confidence: 99%

Contextual Neural Machine Translation Improves Translation of Cataphoric Pronouns

Wong

Maruf²,

Haffari³

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

The advent of context-aware NMT has resulted in promising improvements in the overall translation quality and specifically in the translation of discourse phenomena such as pronouns. Previous works have mainly focused on the use of past sentences as context with a focus on anaphora translation. In this work, we investigate the effect of future sentences as context by comparing the performance of a contextual NMT model trained with the future context to the one trained with the past context. Our experiments and evaluation, using generic and pronoun-focused automatic metrics, show that the use of future context not only achieves significant improvements over the context-agnostic Transformer, but also demonstrates comparable and in some cases improved performance over its counterpart trained on past context. We also perform an evaluation on a targeted cataphora test suite and report significant gains over the contextagnostic Transformer in terms of BLEU.

show abstract

“…Our work is related to adversarial datasets for testing robustness used in Natural Language Processing tasks such as studying gender bias (Zhao et al, 2018;Rudinger et al, 2018;Stanovsky et al, 2019), natural language inference (Glockner et al, 2018) and classification (Wang et al, 2019). Jwalapuram et al (2019) propose a model for pronoun translation evaluation trained on pairs of sentences consisting of the reference and a system output with differing pronouns. However, as Guillou and Hardmeier (2018) point out, this fails to take into account that often there is not a 1:1 correspondence between pronouns in different languages.…”

Section: Coreference Resolution In Machine Translationmentioning

confidence: 99%

ContraCAT: Contrastive Coreference Analytical Templates for Machine Translation

Stojanovski¹,

Krojer²,

Peskov³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Recent high scores on pronoun translation using context-aware neural machine translation have suggested that current approaches work well. ContraPro is a notable example of a contrastive challenge set for English→German pronoun translation. The high scores achieved by transformer models may suggest that they are able to effectively model the complicated set of inferences required to carry out pronoun translation. This entails the ability to determine which entities could be referred to, identify which entity a sourcelanguage pronoun refers to (if any), and access the target-language grammatical gender for that entity. We first show through a series of targeted adversarial attacks that in fact current approaches are not able to model all of this information well. Inserting small amounts of distracting information is enough to strongly reduce scores, which should not be the case. We then create a new template test set Contracat, designed to individually assess the ability to handle the specific steps necessary for successful pronoun translation. Our analyses show that current approaches to context-aware nmt rely on a set of surface heuristics, which break down when translations require real reasoning. We also propose an approach for augmenting the training data, with some improvements.

show abstract

Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite

Cited by 15 publications

References 31 publications

Dynamic Context Selection for Document-level Neural Machine Translation via Reinforcement Learning

Dynamic Context Selection for Document-level Neural Machine Translation via Reinforcement Learning

Contextual Neural Machine Translation Improves Translation of Cataphoric Pronouns

ContraCAT: Contrastive Coreference Analytical Templates for Machine Translation

Contact Info

Product

Resources

About