Advancing natural language processing (NLP) applications of morphologically rich languages with bidirectional encoder representations from transformers (BERT): an empirical case study for Turkish

Abstract

Language model pre-training architectures have demonstrated to be useful to learn language representations. bidirectional encoder representations from transformers (BERT), a recent deep bidirectional self-attention representation from unlabelled text, has achieved remarkable results in many natural language processing (NLP) tasks with fine-tuning. In this paper, we want to demonstrate the efficiency of BERT for a morphologically rich language, Turkish. Traditionally morphologically difficult languages require dense language pre-processing steps in order to model the data to be suitable for machine learning (ML) algorithms. In particular, tokenization, lemmatization or stemming and feature engineering tasks are needed to obtain an efficient data model to overcome data sparsity or high-dimension problems. In this context, we selected five various Turkish NLP research problems as sentiment analysis, cyberbullying identification, text classification, emotion recognition and spam detection from the literature. We then compared the empirical performance of BERT with the baseline ML algorithms. Finally, we found enhanced results compared to base ML algorithms in the selected NLP problems while eliminating heavy pre-processing tasks.

HSI-LFS-BERT: Novel Hybrid Swarm Intelligence Based Linguistics Feature Selection and Computational Intelligent Model for Alzheimer’s Prediction Using Audio Transcript

“…The transformer system in BERT identifies contextual relationships between words in text [32] by using the decoder and encoder part in order to make predictions based on the input data. Encoder-decoder system of the transformer leverages the attention mechanism to attain state-of-the-art effectiveness on the majority of NLP tasks [33]. The input embeddings are generated by adding a positional encoding of each word with a pre-trained embedding vector.…”

Computational Intelligent Models for Alzheimer's Prediction Using Audio Transcript Data

“…Transformers architecture has two key components associated: Encoder and Decoder. The encoder-decoder system leverages the attention mechanism to attain state-of-the-art effectiveness on the majority of natural language-based activities [45].…”

PerAnSel: A Novel Deep Neural Network-Based System for Persian Question Answering

“…Moreover, to handle the morphological rich problem of the Persian language, we use the BERT language model. Özçift et al [ 17 ] demonstrated that BERT can overcome the morphological rich problem. The following research questions were explored in this article: Does using a native dataset for answer selection task show better performance than using a translated dataset for the Persian language?…”

“…PerAnSel deploys a transformer-based language model to process sentences with other word orders. The transformer-based model is composed of fully connected neural networks and attention mechanism [ 19 ], which enable it to address the morphologically problem in the Persian language [ 17 ]. In order to address the answer selection problem for the Persian language, we use transformer-based models and sequential models in parallel.…”

“…PerAnSel deploys a transformer-based language model to process sentences with other word orders. The transformer-based model is composed of fully connected neural networks and attention mechanism [ 19 ], which enable it to address the morphologically problem in the Persian language [ 17 ].…”