chrF deconstructed: beta parameters and n-gram weights

Popović, Maja

doi:10.18653/v1/w16-2341

Cited by 40 publications

(34 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3 Motivation for adding word n-grams to CHRF A preliminary experiment on a small set of texts reported in previous work (Popović, 2016) with different target languages and different types of DA 1 shown that for poorly rated sentences, the standard deviations of CHRF and WORDF scores are similar -both metrics assign relatively similar (low) scores. On the other hand, for the sentences with higher human rates, the deviations for CHRF are (much) lower.…”

Section: N-gram Based F-scoresmentioning

confidence: 99%

“…Contrary to RR, the relation between CHRF and DA has still not been investigated systematically. Preliminary experiments in previous work (Popović, 2016) shown that, concerning DA, the main advantage of character-based Fscore CHRF in comparison to word-based F-score WORDF is better correlation for good translations for which WORDF often assigns too low scores.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

chrF++: words helping character n-grams

Popović¹

2017

Proceedings of the Second Conference on Machine Translation

Self Cite

173

View full text Add to dashboard Cite

Character n-gram F-score (CHRF) is shown to correlate very well with human relative rankings of different machine translation outputs, especially for morphologically rich target languages. However, its relation with direct human assessments is not yet clear. In this work, Pearson's correlation coefficients for direct assessments are investigated for two currently available target languages, English and Russian. First, different β parameters (in range from 1 to 3) are re-investigated with direct assessment, and it is confirmed that β = 2 is the optimal option. Then separate character and word n-grams are investigated, and the main finding is that, apart from character n-grams, word 1-grams and 2-grams also correlate rather well with direct assessments. Further experiments show that adding word unigrams and bigrams to the standard CHRF score improves the correlations with direct assessments, though it is still not clear which option is better, unigrams only (CHRF+) or unigrams and bigrams (CHRF++). This should be investigated in future work on more target languages.

show abstract

Section: N-gram Based F-scoresmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

chrF++: words helping character n-grams

Popović¹

2017

Proceedings of the Second Conference on Machine Translation

Self Cite

173

View full text Add to dashboard Cite

show abstract

“…A combination of BLEU (Papineni et al, 2002) and word error rate (WER) (Nießen et al, 2000) is used for tuning the system, because tuning on BLEU only resulted in overly long translations. Performance of all systems are reported in terms of BLEU, character F1-score CHRF1 (Popović, 2016) and WER. Statistical significance tests were conducted using approximate randomization tests (Clark et al, 2011).…”

Section: Systemsmentioning

confidence: 99%

Generating titles for millions of browse pages on an e-Commerce site

Mathur

Ueffing

Leusch

2017

Proceedings of the 10th International Conference on Natural Language Generation

View full text Add to dashboard Cite

We present three approaches to generate titles for browse pages in five different languages, namely English, German, French, Italian and Spanish. These browse pages are structured search pages in an e-commerce domain. We first present a rule-based approach to generate these browse page titles. In addition, we also present a hybrid approach which uses a phrase-based statistical machine translation engine on top of the rule-based system to assemble the best title. For the two languages English and German, we have access to a large amount of rule-based generated and human-curated titles. For these languages, we present an automatic post-editing approach which learns how to post-edit the rule-based titles into curated titles.

show abstract

“…The systems have been tuned towards characterF-1.0 (Popovic, 2015(Popovic, , 2016. We optimize the beam search parameters, using a grid search.…”

Section: Training Detailsmentioning

confidence: 99%

Extending hybrid word-character neural machine translation with multi-task learning of morphological analysis

Grönroos¹,

Virpioja²,

Kurimo³

2017

Proceedings of the Second Conference on Machine Translation

View full text Add to dashboard Cite

This article describes the Aalto University entry to the English-to-Finnish news translation shared task in WMT 2017. Our system is an open vocabulary neural machine translation (NMT) system, adapted to the needs of a morphologically complex target language. The main contributions of this paper are 1) implicitly incorporating morphological information to NMT through multi-task learning, 2) adding an attention mechanism to the character-level decoder, combined with character segmentation of names, and 3) a new overattending penalty to beam search.

show abstract

chrF deconstructed: beta parameters and n-gram weights

Cited by 40 publications

References 3 publications

chrF++: words helping character n-grams

chrF++: words helping character n-grams

Generating titles for millions of browse pages on an e-Commerce site

Extending hybrid word-character neural machine translation with multi-task learning of morphological analysis

Contact Info

Product

Resources

About