A Practical Survey on Faster and Lighter Transformers

Abstract

Recurrent neural networks are effective models to process sequences. However, they are unable to learn long-term dependencies because of their inherent sequential nature. As a solution, Vaswani et al. introduced the Transformer, a model solely based on the attention mechanism that is able to relate any two positions of the input sequence, hence modelling arbitrary long dependencies. The Transformer has improved the state-of-the-art across numerous sequence modelling tasks. However, its effectiveness comes at the expense of a quadratic computational and memory complexity with respect to the sequence length, hindering its adoption. Fortunately, the deep learning community has always been interested in improving the models’ efficiency, leading to a plethora of solutions such as parameter sharing, pruning, mixed-precision, and knowledge distillation. Recently, researchers have directly addressed the Transformer’s limitation by designing lower-complexity alternatives such as the Longformer, Reformer, Linformer, and Performer. However, due to the wide range of solutions, it has become challenging for researchers and practitioners to determine which methods to apply in practice in order to meet the desired trade-off between capacity, computation, and memory. This survey addresses this issue by investigating popular approaches to make Transformers faster and lighter and by providing a comprehensive explanation of the methods’ strengths, limitations, and underlying assumptions.

A Survey of Text Classification With Transformers: How Wide? How Large? How Long? How Accurate? How Expensive? How Safe?

“…Additional research efforts, like the Pythia suite, are providing new tools to analyze LLMs and address this issue [66]. Other recent survey papers such as [67], seek to address the issue of how to apply more efficient transformer methods to NLP tasks. Approaches are grouped together by sparse, factorized attention, and architectural change.…”

“…Approaches are grouped together by sparse, factorized attention, and architectural change. However, [67] concludes there are, ". .…”

Identification of Perceived Challenges in the Green Energy Transition by Turkish Society through Sentiment Analysis

“…This model has enabled researchers to approach textual data with novel methods and has become increasingly popular over time due to its effectiveness in acquiring contextual word representations, leading to numerous studies in this area. Upon reviewing the literature, it is evident that many studies typically focus on various aspects of transformer models, including their architecture, efficiency, computational power, memory efficiency, and the development of fast and lightweight variants [52]. On the other hand, in other studies, various NLP applications have been explored, including visualization of transformers for NLP [53], examination of pre-training methods used in transformer models [54], usage of transformers for text summarization tasks [55], application of transformer models for detecting different sentiment levels from text-based data [56], and using transformers for extracting useful information from large datasets [57].…”

Focus Your Attention (with Adaptive IIR Filters)

“…Transformers (Vaswani et al, 2017) have emerged as highly effective models for various tasks, but their widespread adoption has been limited by the quadratic cost of the self-attention mechanism and poor performance on long-range tasks. Researchers have pursued diverse approaches to overcome this challenge and to create efficient transformer architectures (Fournier et al, 2021;. From the perspective of efficiency, techniques such as sparse attention , low-rank attention (Wang et al, 2020;Winata et al, 2020), kernel-based attention (Choromanski et al, 2020), recurrent mechanisms (Hutchins et al, 2022;Dai et al, 2019), and efficient IO-awareness-based implementation (Dao et al, 2022a) proved efficient.…”