Overview of the Second BUCC Shared Task: Spotting Parallel
            Sentences in Comparable Corpora

Zweigenbaum, Pierre; Sharoff, Serge; Rapp, Reinhard

doi:10.18653/v1/w17-2512

Cited by 78 publications

(83 citation statements)

References 10 publications

(18 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since 2017, the workshop on Building and Using Comparable Corpora (BUCC) is organizing a shared task to evaluate the performance of approaches to mine for parallel sentences in comparable corpora (Zweigenbaum et al, 2018). Table 1 summarizes the available data, and Table 2 the official results.…”

Section: Experimental Evaluation: Bucc Shared Task On Mining Bitextsmentioning

confidence: 99%

Filtering and Mining Parallel Data in a Joint Multilingual Space

Schwenk¹

2018

Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

View full text Add to dashboard Cite

We learn a joint multilingual sentence embedding and use the distance between sentences in different languages to filter noisy parallel data and to mine for parallel data in large news collections. We are able to improve a competitive baseline on the WMT'14 English to German task by 0.3 BLEU by filtering out 25% of the training data. The same approach is used to mine additional bitexts for the WMT'14 system and to obtain competitive results on the BUCC shared task to identify parallel sentences in comparable corpora.The approach is generic, it can be applied to many language pairs and it is independent of the architecture of the machine translation system.

show abstract

Section: Experimental Evaluation: Bucc Shared Task On Mining Bitextsmentioning

confidence: 99%

Filtering and Mining Parallel Data in a Joint Multilingual Space

Schwenk¹

2018

Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

View full text Add to dashboard Cite

show abstract

“…It should be noted that our model is specifically tuned for the task of misalignment detection, whereby parallel corpus mining and corpus cleaning (we refer to the WMT shared task on parallel corpus filtering [27,28] and the BUCC shared task on parallel corpus mining [29]), involve other important factors such as selection of sentences based on fluency and diversity anddetection of language errors, which possibly have a more profound effect on NMT performance, (e.g. [4]).…”

Section: Discussionmentioning

confidence: 99%

Misalignment Detection for Web-Scraped Corpora: A Supervised Regression Approach

et al. 2019

View full text Add to dashboard Cite

To build state-of-the-art Neural Machine Translation (NMT) systems, high-quality parallel sentences are needed. Typically, large amounts of data are scraped from multilingual web sites and aligned into datasets for training. Many tools exist for automatic alignment of such datasets. However, the quality of the resulting aligned corpus can be disappointing. In this paper, we present a tool for automatic misalignment detection (MAD). We treated the task of determining whether a pair of aligned sentences constitutes a genuine translation as a supervised regression problem. We trained our algorithm on a manually labeled dataset in the FR-NL language pair. Our algorithm used shallow features and features obtained after an initial translation step. We showed that both the Levenshtein distance between the target and the translated source, as well as the cosine distance between sentence embeddings of the source and the target were the two most important features for the task of misalignment detection. Using gold standards for alignment, we demonstrated that our model can increase the quality of alignments in a corpus substantially, reaching a precision close to 100%. Finally, we used our tool to investigate the effect of misalignments on NMT performance.

show abstract

“…Sentence alignment was a popular research topic in the early days of statistical MT, but received less attention once standard sentencealigned parallel corpora became available. Interest in low-resource MT has led to a resurgence in data gathering methods (Buck and Koehn, 2016;Zweigenbaum et al, 2018;, but we find limited recent work on bilingual sentence alignment.…”

Section: Introductionmentioning

confidence: 94%

Vecalign: Improved Sentence Alignment in Linear Time and Space

Thompson¹,

Koehn²

2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

We introduce Vecalign, a novel bilingual sentence alignment method which is linear in time and space with respect to the number of sentences being aligned and which requires only bilingual sentence embeddings. On a standard German-French test set, Vecalign outperforms the previous state-of-the-art method (which has quadratic time complexity and requires a machine translation system) by 5 F 1 points. It substantially outperforms the popular Hunalign toolkit at recovering Bible verse alignments in medium-to low-resource language pairs, and it improves downstream MT quality by 1.7 and 1.6 BLEU in Sinhala→English and Nepali→English, respectively, compared to the Hunalign-based Paracrawl pipeline.

show abstract

Overview of the Second BUCC Shared Task: Spotting Parallel Sentences in Comparable Corpora

Cited by 78 publications

References 10 publications

Filtering and Mining Parallel Data in a Joint Multilingual Space

Filtering and Mining Parallel Data in a Joint Multilingual Space

Misalignment Detection for Web-Scraped Corpora: A Supervised Regression Approach

Vecalign: Improved Sentence Alignment in Linear Time and Space

Contact Info

Product

Resources

About