Evaluating the morphological competence of Machine Translation Systems

Burlot, Franck; Yvon, François

doi:10.18653/v1/w17-4705

Cited by 69 publications

(50 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The corpus generation scheme can be compared with Burlot and Yvon (2017) in the sense that various morpho-syntactic/semantic features are taken into account. However, here we focus more on making the template sentences help discern the gender bias regarding the translation of genderneutral pronouns.…”

Section: Corpus Generationmentioning

confidence: 99%

On Measuring Gender Bias in Translation of Gender-neutral Pronouns

Cho¹,

Kim²,

Kim³

et al. 2019

Proceedings of the First Workshop on Gender Bias in Natural Language Processing

View full text Add to dashboard Cite

Ethics regarding social bias has recently thrown striking issues in natural language processing. Especially for gender-related topics, the need for a system that reduces the model bias has grown in areas such as image captioning, content recommendation, and automated employment. However, detection and evaluation of gender bias in the machine translation systems are not yet thoroughly investigated, for the task being cross-lingual and challenging to define. In this paper, we propose a scheme for making up a test set that evaluates the gender bias in a machine translation system, with Korean, a language with genderneutral pronouns. Three word/phrase sets are primarily constructed, each incorporating positive/negative expressions or occupations; all the terms are gender-independent or at least not biased to one side severely. Then, additional sentence lists are constructed concerning formality of the pronouns and politeness of the sentences. With the generated sentence set of size 4,236 in total, we evaluate gender bias in conventional machine translation systems utilizing the proposed measure, which is termed here as translation gender bias index (TGBI). The corpus and the code for evaluation is available on-line 1 .

show abstract

Section: Corpus Generationmentioning

confidence: 99%

On Measuring Gender Bias in Translation of Gender-neutral Pronouns

Cho¹,

Kim²,

Kim³

et al. 2019

Proceedings of the First Workshop on Gender Bias in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…Other linguistic phenomena and analysis axes have gathered significant attention in NMT evaluation studies, including anaphora resolution (Hardmeier et al, 2014;Voita et al, 2018) and pronoun translation (Guillou and Hardmeier, 2016), modality (Baker et al, 2012), ellipsis and deixis (Voita et al, 2019), word sense disambiguation (Tang et al, 2018), and morphological competence (Burlot and Yvon, 2017). Nevertheless, the last comprehensive study of the effect of negation in MT pertains to older, phrase-based models (Fancellu and Webber, 2015).…”

Section: Introductionmentioning

confidence: 99%

It’s not a Non-Issue: Negation as a Source of Error in Machine Translation

Hossain¹,

Anastasopoulos²,

Blanco³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

As machine translation (MT) systems progress at a rapid pace, questions of their adequacy linger. In this study we focus on negation, a universal, core property of human language that significantly affects the semantics of an utterance. We investigate whether translating negation is an issue for modern MT systems using 17 translation directions as test bed. Through thorough analysis, we find that indeed the presence of negation can significantly impact downstream quality, in some cases resulting in quality reductions of more than 60%. We also provide a linguistically motivated analysis that directly explains the majority of our findings. We release our annotations and code to replicate our analysis here: https://github.com/ mosharafhossain/negation-mt.

show abstract

“…Some of the above work has specifically targeted the differences in performance between NMT and SMT (Burlot and Yvon, 2017;Sennrich, 2017). There are also other types of error analysis targeting this difference, e.g.…”

Section: Related Workmentioning

confidence: 99%

Discourse-Related Language Contrasts in English-Croatian Human and Machine Translation

Šoštarić¹,

Hardmeier²,

Stymne³

2018

Proceedings of the Third Conference on Machine Translation: Research Papers

View full text Add to dashboard Cite

We present an analysis of a number of coreference phenomena in English-Croatian human and machine translations. The aim is to shed light on the differences in the way these structurally different languages make use of discourse information and provide insights for discourse-aware machine translation system development. The phenomena are automatically identified in parallel data using annotation produced by parsers and word alignment tools, enabling us to pinpoint patterns of interest in both languages. We make the analysis more fine-grained by including three corpora pertaining to three different registers. In a second step, we create a test set with the challenging linguistic constructions and use it to evaluate the performance of three MT systems. We show that both SMT and NMT systems struggle with handling these discourse phenomena, even though NMT tends to perform somewhat better than SMT. By providing an overview of patterns frequently occurring in actual language use, as well as by pointing out the weaknesses of current MT systems that commonly mistranslate them, we hope to contribute to the effort of resolving the issue of discourse phenomena in MT applications.

show abstract

Evaluating the morphological competence of Machine Translation Systems

Cited by 69 publications

References 28 publications

On Measuring Gender Bias in Translation of Gender-neutral Pronouns

On Measuring Gender Bias in Translation of Gender-neutral Pronouns

It’s not a Non-Issue: Negation as a Source of Error in Machine Translation

Discourse-Related Language Contrasts in English-Croatian Human and Machine Translation

Contact Info

Product

Resources

About