Leveraging Multilingual News Websites for Building a Kurdish Parallel Corpus

Abstract

Machine translation has been a major motivation of development in natural language processing. Despite the burgeoning achievements in creating more efficient machine translation systems, thanks to deep learning methods, parallel corpora have remained indispensable for progress in the field. In an attempt to create parallel corpora for the Kurdish language, in this article, we describe our approach in retrieving potentially alignable news articles from multi-language websites and manually align them across dialects and languages based on lexical similarity and transliteration of scripts. We present a corpus containing 12,327 translation pairs in the two major dialects of Kurdish, Sorani and Kurmanji. We also provide 1,797 and 650 translation pairs in English-Kurmanji and English-Sorani. The corpus is publicly available under the CC BY-NC-SA 4.0 license. 1

ParsText: A Digraphic Corpus for Tajik-Farsi Transliteration

“…For Jordanian Arabic, Talafha et al ( 2021) created a dataset in Arabic and a non-standard romanization known as Arabizi. Ahmadi et al (2022) compiled a corpus of Kurdish news articles written in the Sorani (Arabic-based) and Kurmanji (Latinbased) orthographies. More recently, Gow-Smith et al (2022) reconstructed part of a 16th-century Scottish Gaelic manuscript in modern orthography.…”

The Kurdish Language Corpus: State of the Art

“…But the system has been tested the tokenization methods for developing a neural machine translation. The authors in [26] described an approach for retrieving potentially-alignable articles of news from websites dependent on lexical similarity and transliteration of scripts. This corpus included 12,327 translation pairs for dialects Sorani, and Kurmanji as well as including 1,797 and 650 translation pairs in English-Sorani and English-Kurmanji.…”