The NLP Cookbook: Modern Recipes for Transformer Based Deep Learning Architectures

Abstract

In recent years, Natural Language Processing (NLP) models have achieved phenomenal success in linguistic and semantic tasks like text classification, machine translation, cognitive dialogue systems, information retrieval via Natural Language Understanding (NLU), and Natural Language Generation (NLG). This feat is primarily attributed due to the seminal Transformer architecture, leading to designs such as BERT, GPT (I, II, III), etc. Although these large-size models have achieved unprecedented performances, they come at high computational costs. Consequently, some of the recent NLP architectures have utilized concepts of transfer learning, pruning, quantization, and knowledge distillation to achieve moderate model sizes while keeping nearly similar performances as achieved by their predecessors. Additionally, to mitigate the data size challenge raised by language models from a knowledge extraction perspective, Knowledge Retrievers have been built to extricate explicit data documents from a large corpus of databases with greater efficiency and accuracy. Recent research has also focused on superior inference by providing efficient attention to longer input sequences. In this paper, we summarize and examine the current state-of-the-art (SOTA) NLP models that have been employed for numerous NLP tasks for optimal performance and efficiency. We provide a detailed understanding and functioning of the different architectures, a taxonomy of NLP designs, comparative evaluations, and future directions in NLP.

A Narrative Literature Review of Natural Language Processing Applied to the Occupational Exposome

“…This includes but is not limited to transfer learning [ 76 ] or adversarial learning [ 77 ], which include a variety of neural networks structures or knowledge graphs that have been at the core of NLP research. This also includes Transformer-based methods [ 78 ] (e.g., large pre-trained language models such as BERT [ 79 ]), which have made a significant impact on the field of NLP over recent years and could prove to be useful in NLP for occupational exposure research. This type of deep learning method is based on attention [ 80 ], which has been shown to improve results in a variety of other domains that have utilised NLP (e.g., healthcare).…”

A Genetic Algorithm-Based Approach to Support Forming Multiple Scrum Project Teams

“…However, computational algorithms have difficulty in recognizing different text sentences as the same [47], [48]. Natural Language Processing approaches [49]- [51] have been proposed over the years. However, they depend on the quality and completeness of the information presented in the text.…”

The HoPE Model Architecture: a Novel Approach to Pregnancy Information Retrieval Based on Conversational Agents

“…Self-attention enables the look-up of remaining input words at various positions to determine the significance of the currently processed word. This is done for all input words to improve the encoding and context understanding of all words [ 96 ]. We present an illustration of this architecture in Fig.…”