Creating language resources for under-resourced languages: methodologies, and experiments with Arabic

Abstract

Language resources are important for those working on computational methods to analyse and study languages. These resources are needed to help advancing the research in fields such as natural language processing, machine learning, information retrieval and text analysis in general. We describe the creation of useful resources for languages that currently lack them, taking resources for Arabic summarisation as a case study. We illustrate three different paradigms for creating language resources, namely: (1) using crowdsourcing to produce a small resource rapidly and relatively cheaply; (2) translating an existing gold-standard dataset, which is relatively easy but potentially of lower quality; and (3) using manual effort with appropriately skilled human participants to create a resource that is more expensive but of high quality. The last of these was used as a test collection for TAC-2011. An evaluation of the resources is also presented.The current paper describes and extends the resource creation activities and evaluations that underpinned experiments and findings that have previously appeared as an LREC workshop paper (El-Haj et al 2010), a student conference paper (El-Haj et al 2011b), and a description of a multilingual summarisation pilot (El-Haj et al 2011c;.

Associative Root–Pattern Data and Distribution in Arabic Morphology

“…• A templatic pattern in the form of consonant-vowel arrangement picturing the morpho-phonetic structure of a word form completing possible phonetic, syntactic, and semantic information 6 .…”

“…Examples include as follows: 6 In the statistical analysis of each pattern, syntactical suffixes are also considered to form multiple phonetic patterns conveying additional syntactical information at the end of a pattern.…”

“…For comparison, there is even in an earlier study [5], a report of 19 Arabic corpora ranging in size from 1-3 billion words. In spite of that, Arabic is still poor in resources in terms of quality and availability [6,7]. By contrast to the current available corpora, our approach emerges from a different perspective of building the APRoPAT 2 Corpus in the following senses:…”

Towards building a standard dataset for Arabic keyphrase extraction evaluation

“…[10] showed that the quality of Arabic workers available in Mechanical Turk was not satisfying enough for POS (Part Of Speech) tagging or grammatical case annotation. However, [11] used with satisfaction the same platform for building an Arabic corpus for the much easier task of text summarization.…”

“…The document collection we used contains 160 documents selected from four general purpose, freely available corpora: 46 documents from Arabic Newspapers Corpus (ANC) [12], 53 from Corpus of Contemporary Arabic (CCA) [13], 31 from Essex Arabic Summaries Corpus (EASC) [14], [11], and 30 from Open Source Arabic Corpora (OSAC) [15].…”