Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?

Abstract

Transformers have sprung up in the field of computer vision. In this work, we explore whether the core self-attention module in Transformer is the key to achieving excellent performance in image recognition. To this end, we build an attention-free network called sMLPNet based on the existing MLP-based vision models. Specifically, we replace the MLP module in the token-mixing step with a novel sparse MLP (sMLP) module. For 2D image tokens, sMLP applies 1D MLP along the axial directions and the parameters are shared among rows or columns. By sparse connection and weight sharing, sMLP module significantly reduces the number of model parameters and computational complexity, avoiding the common over-fitting problem that plagues the performance of MLP-like models. When only trained on the ImageNet-1K dataset, the proposed sMLPNet achieves 81.9% top-1 accuracy with only 24M parameters, which is much better than most CNNs and vision Transformers under the same model size constraint. When scaling up to 66M parameters, sMLPNet achieves 83.4% top-1 accuracy, which is on par with the state-of-the-art Swin Transformer. The success of sMLPNet suggests that the self-attention mechanism is not necessarily a silver bullet in computer vision. The code and models are publicly available at https://github.com/microsoft/SPACH.

Piecewise linear neural networks and deep learning

“…Can we find such a PWLNN by explicitly seeking a shallow PWLNN or implicitly regularizing the learning of a PWL-DNN? What are the differences and relations between PWLNNs and other kinds of NNs that address locally-dominant features 195 ?…”

TAMFN: Time-Aware Attention Multimodal Fusion Network for Depression Detection

“…The module first extracts the long-term context dependencies of each modality using LSTM, and then further extracts the temporal vectors of each modality. After that, the sparse MLP [57] is used to mix the information of the temporal importance of the two modalities to obtain the attention vector with interaction information. Finally, the attention vector with interaction information is used to guide multimodal feature fusion.…”

“…2) The temporal features of the two modalities are concatenated to obtain the concatenated vector Concat vector. To interact with the timing information of the two modalities, we mix the information in the vertical and horizontal directions of the concatenated vector Concat vector through weight sharing and sparse connection in the sparse MLP [57], respectively, to obtain the mixed attention vector x mix.…”

GPA-TUNet: Transformer and GPA Attention Co-Encoder for Medical Image Segmentation

“…sMLP block. Chuanxin Tang et al proposed Sparse MLP (sMLP) [21] based on the MLP-based vision model, replacing the MLP module in the token-mixing step with a new sMLP module. For a 2D image, sMLP applies 1D MLP along the image height and width, so the parameters are shared between rows or columns.…”

“…In order to solve these problems, we design a new attention mechanism GPA and cite the Sparse-MLP (sMLP) proposed by Chuanxin Tang et al [21]. We combine GPA with Transformer as encoder.…”