CharacTer: Translation Edit Rate on Character Level

Wang, Weiyue; Peter, J. Dinesh; Rosendahl, Hendrik; Ney, Hermann

doi:10.18653/v1/w16-2342

Cited by 89 publications

(72 citation statements)

References 11 publications

(18 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, we have a set of systems that are optimized in order to improve target morphology. The automatic scores of the systems submitted at WMT'17 8 are in Table 4 where we report BLEU, BEER (Stanojević and Sima'an, 2014) and CharacTER (Wang et al, 2016). 9 We also computed a morphology accuracy for these systems.…”

Section: Methodsmentioning

confidence: 99%

Evaluating the morphological competence of Machine Translation Systems

Burlot¹,

Yvon²

2017

Proceedings of the Second Conference on Machine Translation

View full text Add to dashboard Cite

While recent changes in Machine Translation state-of-the-art brought translation quality a step further, it is regularly acknowledged that the standard automatic metrics do not provide enough insights to fully measure the impact of neural models. This paper proposes a new type of evaluation focused specifically on the morphological competence of a system with respect to various grammatical phenomena. Our approach uses automatically generated pairs of source sentences, where each pair tests one morphological contrast. This methodology is used to compare several systems submitted at WMT'17 for English into Czech and Latvian.

show abstract

Section: Methodsmentioning

confidence: 99%

Evaluating the morphological competence of Machine Translation Systems

Burlot¹,

Yvon²

2017

Proceedings of the Second Conference on Machine Translation

View full text Add to dashboard Cite

show abstract

“…BLEU scores for English-Czech Results are in Table 5, where we provide, in addition to BLEU, scores computed by BEER (Stanojević and Sima'an, 2014) and CharacTER (Wang et al, 2016). These two metrics proved to be more adapted to MRLs by Bojar et al (2016).…”

Section: Small Systemmentioning

confidence: 99%

Learning Morphological Normalization for Translation from and into Morphologically Rich Languages

Burlot¹,

Yvon²

2017

The Prague Bulletin of Mathematical Linguistics

View full text Add to dashboard Cite

When translating between a morphologically rich language (MRL) and English, word forms in the MRL often encode grammatical information that is irrelevant with respect to English, leading to data sparsity issues. This problem can be mitigated by removing from the MRL irrelevant information through normalization. Such preprocessing is usually performed in a deterministic fashion, using hand-crafted rules and yielding suboptimal representations. We introduce here a simple way to automatically compute an appropriate normalization of the MRL and show that it can improve machine translation in both directions.

show abstract

“…CHARACTER (Wang et al, 2016), identical to the 2016 setup, is a character-level metric inspired by the commonly applied translation edit rate (TER). It is defined as the minimum number of character edits required to adjust a hypothesis, until it completely matches the reference, normalized by the length of the hypothesis sentence.…”

Section: Charactermentioning

confidence: 99%

Results of the WMT17 Metrics Shared Task

Bojar¹,

Graham²,

Kamran³

2017

Proceedings of the Second Conference on Machine Translation

103

View full text Add to dashboard Cite

This paper presents the results of the WMT17 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in the WMT17 news translation task and Neural MT training task. We collected scores of 14 metrics from 8 research groups. In addition to that, we computed scores of 7 standard metrics (BLEU, SentBLEU, NIST, WER, PER, TER and CDER) as baselines. The collected scores were evaluated in terms of system-level correlation (how well each metric's scores correlate with WMT17 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in judging the quality of a particular sentence).This year, we build upon two types of manual judgements: direct assessment (DA) and HUME manual semantic judgements.

show abstract

CharacTer: Translation Edit Rate on Character Level

Cited by 89 publications

References 11 publications

Evaluating the morphological competence of Machine Translation Systems

Evaluating the morphological competence of Machine Translation Systems

Learning Morphological Normalization for Translation from and into Morphologically Rich Languages

Results of the WMT17 Metrics Shared Task

Contact Info

Product

Resources

About