Integrated Parallel Sentence and Fragment Extraction from Comparable Corpora

Chu, Chenhui; Nakazawa, Toshiaki; Kurohashi, Sadao

doi:10.1145/2833089

Cited by 18 publications

(14 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We collected all the publicly available, parallel Chinese-Japanese corpora we could find, and made it available to participants as the existing parallel. These include Global Voices, News Commentary, and Ubuntu corpora from OPUS Tiedemann (2012); OpenSubtitles (Lison and Tiedemann, 2016); TED talks (Dabre and Kurohashi, 2017); Wikipedia (Chu et al, 2014(Chu et al, , 2015; Wiktionary.org; and WikiMatrix (Schwenk et al, 2019). We also collected parallel sentences from Tatoeba.org, released under a CC-BY License.…”

Section: Parallel Training Datamentioning

confidence: 99%

Findings of the Iwslt 2020 Evaluation Campaign

Ansari¹,

Axelrod²,

Bach³

et al. 2020

Proceedings of the 17th International Conference on Spoken Language Translation

View full text Add to dashboard Cite

The evaluation campaign of the International Conference on Spoken Language Translation (IWSLT 2020) featured this year six challenge tracks: (i) Simultaneous speech translation, (ii) Video speech translation, (iii) Offline speech translation, (iv) Conversational speech translation, (v) Open domain translation, and (vi) Non-native speech translation. A total of 30 teams participated in at least one of the tracks. This paper introduces each track's goal, data and evaluation metrics, and reports the results of the received submissions.

show abstract

Section: Parallel Training Datamentioning

confidence: 99%

Findings of the Iwslt 2020 Evaluation Campaign

Ansari¹,

Axelrod²,

Bach³

et al. 2020

Proceedings of the 17th International Conference on Spoken Language Translation

View full text Add to dashboard Cite

show abstract

“…They attempt to find the translations in tweets instead of translating the texts. [8] extract both parallel sentences and fragments from comparable corpora of Chinese-Japanese Wikipedia to improve statistical MT. [12] apply a domain-biased parallel data collection and a structured methodology to obtain English-Hindi parallel data.…”

Section: Related Workmentioning

confidence: 99%

Parallel Data Extraction using Word Embeddings

Lohar¹,

Way²

2020

Computer Science &Amp; Information Technology (CS &Amp; IT)

View full text Add to dashboard Cite

Building a robust MT system requires a sufficiently large parallel corpus to be available as training data. In this paper, we propose to automatically extract parallel sentences from comparable corpora without using any MT system or even any parallel corpus at all. Instead, we use crosslingual information retrieval (CLIR), average word embeddings, text similarity and a bilingual dictionary, thus saving a significant amount of time and effort as no MT system is involved in this process. We conduct experiments on two different kinds of data: (i) formal texts from news domain, and (ii) user-generated content (UGC) from hotel reviews. The automatically extracted sentence pairs are then added to the already available parallel training data and the extended translation models are built from the concatenated data sets. Finally, we compare the performance of our new extended models against the baseline models built from the available data. The experimental evaluation reveals that our proposed approach is capable of improving the translation outputs for both the formal texts and UGC.

show abstract

“…In recent years, there have been several approaches developed for obtaining parallel sentences or fragments from non-parallel data [8], [9], such as comparable data [8], [10], [11], [12] and quasi-comparable data [13] to make contributions to SMT. Parallel corpora contain parallel sentences, i.e., sentences which are translations of each other.…”

Section: Related Workmentioning

confidence: 99%

Inflating a Small Parallel Corpus into a Large Quasi-parallel Corpus Using Monolingual Data for Chinese-Japanese Machine Translation

Yang

Shen

Lepage

2017

Journal of Information Processing

View full text Add to dashboard Cite

Increasing the size of parallel corpora for less-resourced language pairs is essential for machine translation (MT). To address the shortage of parallel corpora between Chinese and Japanese, we propose a method to construct a quasi-parallel corpus by inflating a small amount of Chinese-Japanese corpus, so as to improve statistical machine translation (SMT) quality. We generate new sentences using analogical associations based on large amounts of monolingual data and a small amount of parallel data. We filter over-generated sentences using two filtering methods: one based on BLEU and the second one based on N-sequences. We add the obtained aligned quasi-parallel corpus to a small parallel Chinese-Japanese corpus and perform SMT experiments. We obtain significant improvements over a baseline system.

show abstract

Integrated Parallel Sentence and Fragment Extraction from Comparable Corpora

Cited by 18 publications

References 25 publications

Findings of the Iwslt 2020 Evaluation Campaign

Findings of the Iwslt 2020 Evaluation Campaign

Parallel Data Extraction using Word Embeddings

Inflating a Small Parallel Corpus into a Large Quasi-parallel Corpus Using Monolingual Data for Chinese-Japanese Machine Translation

Contact Info

Product

Resources

About