Alignment of comparable documents: Comparison of similarity measures on French–English–Arabic data

Abstract

The objective, in this article, is to address the issue of the comparability of documents, which are extracted from different sources and written in different languages. These documents are not necessarily translations of each other. This material is referred as multilingual comparable corpora. These language resources are useful for multilingual natural language processing applications, especially for low-resourced language pairs. In this paper, we collect different data in Arabic, English, and French. Two corpora are built by using available hyperlinks for Wikipedia and Euronews. Euronews is an aligned multilingual (Arabic, English, and French) corpus of 34k documents collected from Euronews website. A more challenging issue is to build comparable corpus from two different and independent media having two distinct editorial lines, such as British Broadcasting Corporation (BBC) and Al Jazeera (JSC). To build such corpus, we propose to use the Cross-Lingual Latent Semantic approach. For this purpose, documents have been harvested from BBC and JSC websites for each month of the years 2012 and 2013. The comparability is calculated for each Arabic–English couple of documents of each month. This automatic task is then validated by hand. This led to a multilingual (Arabic–English) aligned corpus of 305 pairs of documents (233k English words and 137k Arabic words). In addition, a study is presented in this paper to analyze the performance of three methods of the literature allowing to measure the comparability of documents on the multilingual reference corpora. A recall at rank 1 of 50.16 per cent is achieved with the Cross-lingual LSI approach for BBC–JSC test corpus, while the dictionary-based method reaches a recall of only 35.41 per cent.

An Experimental Study with Fuzzy-Wuzzy (Partial Ratio) for Identifying the Similarity between English and French Languages for Plagiarism Detection

“…Fig. 4 illustrates the data from Table V to VIII which contains many-to-many mappings of [1,6], [3,10], [5,7] and [4,6] French Documents and different English documents. It shows that the accuracy of Fuzzy-Wuzzy (Partial Ratio) technique is more than the accuracy of remaining presented techniques.…”

“…The basis for this estimation could be bilingual dictionaries or digital techniques, for example, latent semantic indexing (LSI) [16]. To find similar Arabic/English documents in two ways, LSI is used [1]. Monolingual: the first way is to translate the English article into Arab and then map it into space in the Arabic language LSI [10].…”

A Privacy-Preserving Multilingual Comparable Corpus Construction Method in Internet of Things

“…And, the privacy and security of these data sources should be ensured. The relevant research on provable construction methods of privacy-preserving comparable corpora have been carried out at home and abroad [8], with the following three main methods: that based on word frequency distribution [28], that based on feature distribution, and that based on cross-lingual retrieval; e.g., D. Langlos et al [29] used a cross-page semantic feature approach to obtain Arabic, English and French data and constructed a trilingual corpus, but the capacity of the corpus was comparably small, with only 305 comparable corpus pairs, and they ignored the privacy-preserving issue. The web-based construction method is a basic resource of comparable corpora, which mainly includes news websites; e.g., Yuan Wei [30] built a Chinese-Russian comparable news corpus by acquiring news corpora through the Xinhua website.…”

“…Many previous studies have found the close relationship between IoT data sharing and data privacy protection. Currently, the focus on data privacy protection is mainly in specific areas, such as the data analysis of patient conditions in the medical field [37], industry information [4] and data protection in the railway transportation sector [29]. There is relatively less coverage of privacy protection concerning multilingual data based on IoT news.…”

A Fine-Grained Multilingual Analysis Based on the Appraisal Theory: Application to Arabic and English Videos

“…In other words, we have to align two texts one is in Arabic and the second is in English. Several works, on multilingual comparability, have been proposed by the international community [10], [22], [9], [3], [13]. Overall, they concern documents harvested from social networks, Wikipedia, etc.…”