The development of a sepedi text generation model using long-short term memory

Moila, Mercy M.; Modipa, Thipe I.

doi:10.1145/3415088.3415114

Cited by 2 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, they are robust when generating local coherent text. Therefore, several papers such as [6]- [8], [71]- [76] discuss various deep learning RNN algorithms in light to this shortcoming. Improved versions of RNN, namely, LSTM and GRU are implemented in an attempt to solve long-term dependencies in text.…”

Section: Machine Learning Algorithms For Text Generationmentioning

confidence: 99%

Adversarial Machine Learning in Text Processing: A Literature Survey

et al. 2022

View full text Add to dashboard Cite

Machine learning algorithms represent the intelligence that controls many information systems and applications around us. As such, they are targeted by attackers to impact their decisions. Text created by machine learning algorithms has many types of applications, some of which can be considered malicious especially if there is an intention to present machine-generated text as human-generated. In this paper, we surveyed major subjects in adversarial machine learning for text processing applications. Unlike adversarial machine learning in images, text problems and applications are heterogeneous. Thus, each problem can have its own challenges. We focused on some of the evolving research areas such as: malicious versus genuine text generation metrics, defense against adversarial attacks, and text generation models and algorithms. Our study showed that as applications of text generation will continue to grow in the near future, the type and nature of attacks on those applications and their machine learning algorithms will continue to grow as well. Literature survey indicated an increasing trend in using pre-trained models in machine learning. Word/sentence embedding models and transformers are examples of those pre-trained models. Adversarial models may utilize same or similar pre-trained models as well. In another trend related to text generation models, literature showed effort to develop universal text perturbations to be used in both black-and whitebox attack settings. Literature showed also using conditional GANs to create latent representation for writing types. This usage will allow for a seamless lexical and grammatical transition between various writing styles. In text generation metrics, research trends showed developing successful automated or semi-automated assessment metrics that may include human judgement. Literature showed also research trends of designing and developing new memory models that increase performance and memory utilization efficiency without validating real-time constraints. Many research efforts evaluate different defense model approaches and algorithms. Researchers evaluated different types of targeted attacks, and methods to distinguish human versus machine generated text.

show abstract

Section: Machine Learning Algorithms For Text Generationmentioning

confidence: 99%

Adversarial Machine Learning in Text Processing: A Literature Survey

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Different text generation models have been developed for many resourced languages (Islam et al, 2019). There are few models for most under-resourced languages such as Sepedi (Moila & Modipa, 2020). A model that generates text that cannot be differentiated from human-generated or machine-generated text would be useful for text corpus creation.…”

Section: Introductionmentioning

confidence: 99%

The Analysis of a GPT-based Sepedi Text Generation Model

Moila¹,

Modipa²,

Manamela³

2022

ICONIC

View full text Add to dashboard Cite

Text generation is defined as a component of natural language processing that makes use of computational linguistics techniques to produce text that cannot be distinguished from human written text. This study aims to develop and analyse a Generative Pre-Trained Transformer 2 (GPT-2) language model to generate Sepedi phrases. The under-resourced Sepedi language is regarded as a disjunctive language. The Sepedi language orthographic representation presents challenges and has limited resources. The GPT-2 transformer requires large datasets, as well as state-of-the-art computational resources. The unstructured National Centre for Human Language Technology (NCHLT) Sepedi text dataset was used. The text generation model developed with the small dataset managed to get the lowest loss value of 2.36. The output text generated using this model produces a text that is syntactically correct with instances of grammatical errors. The model performed better than previously developed Sepedi text generation models by using transformer-based technique.

show abstract

The development of a sepedi text generation model using long-short term memory

Cited by 2 publications

References 7 publications

Adversarial Machine Learning in Text Processing: A Literature Survey

Adversarial Machine Learning in Text Processing: A Literature Survey

The Analysis of a GPT-based Sepedi Text Generation Model

Contact Info

Product

Resources

About