Projecting Named Entity Tags From a Resource Rich Language to a Resource Poor Language

Abstract

Named Entities (NE) are the prominent entities appearing in textual documents. Automatic classification of NE in a textual corpus is a vital process in Information Extraction and Information Retrieval research. Named Entity Recognition (NER) is the identification of words in text that correspond to a pre-defined taxonomy such as person, organization, location, date, time, etc. This article focuses on the person (PER), organization (ORG) and location (LOC) entities for a Malay journalistic corpus of terrorism. A projection algorithm, using the Dice Coefficient function and bigram scoring method with domain-specific rules, is suggested to map the NE information from the English corpus to the Malay corpus of terrorism. The English corpus is the translated version of the Malay corpus. Hence, these two corpora are treated as parallel corpora. The method computes the string similarity between the English words and the list of available lexemes in a pre-built lexicon that approximates the best NE mapping. The algorithm has been effectively evaluated using our own terrorism tagged corpus; it achieved satisfactory results in terms of precision, recall, and F-measure. An evaluation of the selected open source NER tool for English is also presented.

Low-resource Languages: A Review of Past Work and Future Challenges

“…NER and NET Both tasks are often jointly learnt as a classification task which includes a category for wrods that are not named entities. Projection has been widely used in this task, either to learn a model on a HRL and applying it to a LRL (Zamin, 2020), or to train a classifier on the projected annotations from a HRL (Yarowsky et al, 2001). The early solutions to solve this task mainly involved Hidden Markovian Models (HMM), but they fail to handle named entity phrases and cannot model non-local dependencies along the sentence (Yarowsky et al, 2001).…”

“…Future work As (Zamin, 2020) reported, performance of NER systems is highly currently domain-specific ; some papers incentivize designing cross-domain techniques. The literature mentioned that conceiving a new metrics to compare cross-lingual embedding is promising avenue to XEL.…”

Named Entity Recognition in Low-resource Languages using Cross-lingual distributional word representation

“…Introduced in 2001, annotation projection is the process of transferring annotations from one language to another language [17]. For instance, in order to build the first named entity annotated corpus for Malay, named entity tags have been projected from English (resource-rich) to Malay (low-resource) language with an F-score of 90% on their test set [18]. Based on a Malay-English align corpus, their method can be resume is three steps: they have firstly annotated the English part of the corpus using UIUC named entity tagger [14].…”