Viability of Neural Networks for Core Technologies for Resource-Scarce Languages

Abstract

In this paper, the viability of neural network implementations of core technologies (the focus of this paper is on text technologies) for 10 resource-scarce South African languages is evaluated. Neural networks are increasingly being used in place of other machine learning methods for many natural language processing tasks with good results. However, in the South African context, where most languages are resource-scarce, very little research has been done on neural network implementations of core language technologies. In this paper, we address this gap by evaluating neural network implementations of four core technologies for ten South African languages. The technologies we address are part of speech tagging, named entity recognition, compound analysis and lemmatization. Neural architectures that performed well on similar tasks in other settings were implemented for each task and the performance was assessed in comparison with currently used machine learning implementations of each technology. The neural network models evaluated perform better than the baselines for compound analysis, are viable and comparable to the baseline on most languages for POS tagging and NER, and are viable, but not on par with the baseline, for Afrikaans lemmatization.

A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios

“…For part-of-speech (POS) tagging, Garrette and Baldridge (2013) limit the time of the annotators to 2 hours resulting in up to 1-2k tokens. Kann et al (2020) study languages that have less than 10k labeled tokens in the Universal Dependency project (Nivre et al, 2020) and Loubser and Puttkammer (2020) report that most available datasets for South African languages have 40-60k labeled tokens.…”

Evaluation of Neural Network Transformer Models for Named-Entity Recognition on Low-Resourced Languages

“…LM-Roberta (XLM-R) is a recent transformer model that has reported state-of-the-art results for Natural Language Processing (NLP) tasks and applications, such as Named-Entity Recognition (NER), Part-of-Speech (POS) tagging, phrase chunking, and Machine Translation (MT) [2], [8]. The NER and POS sequence tagging tasks have been extensively researched [1]- [6], [8], [9]. However, within the past few years, the introduction of new Deep Learning (DL) transformer model architectures such as XLM-R, Multilingual Bidirectional Encoder Representations from Transformers (M-BERT) and Cross-Lingual Language Model (XLM) lowers the time needed to train large datasets through greater parallelization [7].…”

“…However, within the past few years, the introduction of new Deep Learning (DL) transformer model architectures such as XLM-R, Multilingual Bidirectional Encoder Representations from Transformers (M-BERT) and Cross-Lingual Language Model (XLM) lowers the time needed to train large datasets through greater parallelization [7]. This allows low-resourced languages to be trained and tested with less effort and resource costs, with state-of-the-art results for sequence tagging tasks [1], [2], [8]. M-BERT as a single language model pre-trained from monolingual corpora performs very well with cross-lingual generalization [10].…”

“…A recent study by [1] researched the "Viability of Neural Networks for Core Technologies for Resource-Scarce Languages". These resource-scarce languages are ten of the 11 official South African (SA) languages, with English being excluded.…”

“…These resource-scarce languages are ten of the 11 official South African (SA) languages, with English being excluded. The languages are considered lowresourced, with Afrikaans (af) being the more resourced of the ten [1], [9]. This recent study looked at sequence tagging (POS tagging and NER) and sequence translation (Lemmatization and Compound Analysis), comparing two Bidirectional Long Short-Term Memory with Auxiliary Loss (bi-LSTM-aux) NN models to a baseline Conditional Random Fields (CRF) model.…”