Survey of transformers and towards ensemble learning using transformers for natural language processing

Abstract

The transformer model is a famous natural language processing model proposed by Google in 2017. Now, with the extensive development of deep learning, many natural language processing tasks can be solved by deep learning methods. After the BERT model was proposed, many pre-trained models such as the XLNet model, the RoBERTa model, and the ALBERT model were also proposed in the research community. These models perform very well in various natural language processing tasks. In this paper, we describe and compare these well-known models. In addition, we also apply several types of existing and well-known models which are the BERT model, the XLNet model, the RoBERTa model, the GPT2 model, and the ALBERT model to different existing and well-known natural language processing tasks, and analyze each model based on their performance. There are a few papers that comprehensively compare various transformer models. In our paper, we use six types of well-known tasks, such as sentiment analysis, question answering, text generation, text summarization, name entity recognition, and topic modeling tasks to compare the performance of various transformer models. In addition, using the existing models, we also propose ensemble learning models for the different natural language processing tasks. The results show that our ensemble learning models perform better than a single classifier on specific tasks. Graphical Abstract

Artificial Intelligence in Newborn Medicine

“…• Model architecture: The GPT models are based on a transformer architecture, which consists of multiple layers of self-attention mechanisms and feedforward neural networks. 79 The number of layers, hidden units, attention heads, and other architectural parameters can vary depending on the size and complexity of the model. • Pre-training data: Models pre-trained on large amounts of text data to learn language representations can be useful.…”

“…There are several alternatives to GPT for NLP tasks, each with its own strengths and weaknesses. 79 Here are some notable alternatives:…”