Neural Architecture Search for Transformers: A Survey

Abstract

Transformer-based Deep Neural Network architectures have gained tremendous interest due to their effectiveness in various applications across Natural Language Processing (NLP) and Computer Vision (CV) domains. These models are the de facto choice in several language tasks, such as Sentiment Analysis and Text Summarization, replacing Long Short Term Memory (LSTM) model. Vision Transformers (ViTs) have shown better model performance than traditional Convolutional Neural Networks (CNNs) in vision applications while requiring significantly fewer parameters and training time. The design pipeline of a neural architecture for a given task and dataset is extremely challenging as it requires expertise in several interdisciplinary areas such as signal processing, image processing, optimization and allied fields. Neural Architecture Search (NAS) is a promising technique to automate the architectural design process of a Neural Network in a data-driven way using Machine Learning (ML) methods. The search method explores several architectures without requiring significant human effort, and the searched models outperform the manually built networks. In this paper, we review Neural Architecture Search techniques, targeting the Transformer model and its family of architectures such as Bidirectional Encoder Representations from Transformers (BERT) and Vision Transformers. We provide an in-depth literature review of approximately 50 state-of-theart Neural Architecture Search methods and explore future directions in this fast-evolving class of problems.

A Comprehensive Survey of Transformers for Computer Vision

“…In contrast, NAS has not yet been fully explored for ViTs. In [160], the authors surveyed several NAS techniques for ViTs. To the best of our knowledge, there are limited studies on the NAS exploration in ViTs [161][162][163][164][165][166], and more attention is needed in the future.…”

Performance Improvement on Traditional Chinese Task-Oriented Dialogue Systems With Reinforcement Learning and Regularized Dropout Technique

“…The cross entropy between 𝑝(𝑥) and 𝑞(𝑥; 𝜃) is given by 𝐻(𝑝, 𝑞) ≡ −𝔼 𝑝 [log 𝑞(𝑥; 𝜃)], (16) where the second equality holds if the number of samples (𝑚) is large enough. Equation (16) shows that maximizing the likelihood in (15) with respect to the parameter 𝜃 is equivalent to minimizing the cross entropy of (13).…”

“…Thus, they can be regarded as a special kind of regularization [12] to improve overfitting due to small datasets. At present, Transformer has become the mainstream architecture of PTMs for NLP tasks [13]. The well-known pre-trained language models BERT, GPT-2 and GPT-3 [14]- [15] are extensions of the Transformer architecture.…”

Hyperspectral upgrade solution for biomicroscope combined with Transformer network to classify infectious bacteria

“…Utilizing this self‐built prototype system, we successfully obtained hyperspectral images of four distinct bacterial pathogens and analyzed their spectral differences. Additionally, inspired by the great achievements of the Transformer network in the field of natural language processing (NLP) and image processing, such as ChatGPT, ViT, and so on [23]. This article is dedicated to extending the network architecture to identify hyperspectral images of infectious pathogens.…”