English-Czech Systems in WMT19: Document-Level Transformer

Abstract

We describe our NMT systems submitted to the WMT19 shared task in English→Czech news translation. Our systems are based on the Transformer model implemented in either Tensor2Tensor (T2T) or Marian framework.We aimed at improving the adequacy and coherence of translated documents by enlarging the context of the source and target. Instead of translating each sentence independently, we split the document into possibly overlapping multi-sentence segments. In case of the T2T implementation, this "documentlevel"-trained system achieves a +0.6 BLEU improvement (p < 0.05) relative to the same system applied on isolated sentences. To assess the potential effect document-level models might have on lexical coherence, we performed a semi-automatic analysis, which revealed only a few sentences improved in this aspect. Thus, we cannot draw any conclusions from this week evidence.

Findings of the 2019 Conference on Machine Translation (WMT19)

“…(no associated paper) CAIRE CUNI Charles University (Popel et al, 2019;Kocmi and Bojar, 2019) and (Kvapilíková et al, 2019)…”

“…and CUNI-TRANSFORMER-T2T2019 (Popel et al, 2019) are trained in the T2T framework following the last year submission (Popel, 2018), but training on WMT19 document-level parallel and monoliongual data. During decoding, each document is split into overlapping multi-sentence segments, where only the "middle" sentences in each segment are used for the final translation.…”

“…(Hokamp et al, 2019) The Aylien research team built a Multilingual NMT system which is trained on all WMT2019 language pairs in all directions, then fine-tuned for a small number of iterations on Gujarati-English data only, including some self-backtranslated data. (no associated paper) CAIRE CUNI Charles University (Popel et al, 2019;Kocmi and Bojar, 2019) and (Kvapilíková et al, 2019) DBMS-KU Kumamoto University, Telkom University, Indonesian Institute of Sciences (Budiwati et al, 2019) 2.5.6 BTRANS Unfortunately, no details are available for this system.…”

CUNI Systems for the Unsupervised News Translation Task in WMT 2019

“…We use the Transformer architecture by Vaswani et al (2017) implemented in Marian framework (Junczys-Dowmunt et al, 2018) to train an NMT model on the synthetic corpus produced by the PBMT model. The model setup, training and decoding hyperparameters are identical to the CUNI Marian systems in English-to-Czech news translation task in WMT19 (Popel et al, 2019), but in this case, due to smaller and noisier training data, we set the dropout between Transformer layers to 0.3. We use 8 Quadro P5000 GPUs with 16GB memory.…”

“…Our other comparison system, Benchmark-TransferEN, was first trained as an English-to-Czech NMT system (see CUNI Transformer Marian for the English-to-Czech news translation task in WMT19 by Popel et al (2019)) and then finetuned for 6 days on the SynthCorpus-noCzechreordered-NER. The vocabulary remained unchanged, it was trained on the English-Czech training corpus.…”