Active Token Mixer

Abstract

This paper presents ActiveMLP, a general MLP-like backbone for computer vision. The three existing dominant network families, i.e., CNNs, Transformers and MLPs, differ from each other mainly in the ways to fuse contextual information into a given token, leaving the design of more effective token-mixing mechanisms at the core of backbone architecture development. In ActiveMLP, we propose an innovative token-mixer, dubbed Active Token Mixer (ATM), to actively incorporate contextual information from other tokens in the global scope into the given one. This fundamental operator actively predicts where to capture useful contexts and learns how to fuse the captured contexts with the original information of the given token at channel levels. In this way, the spatial range of token-mixing is expanded and the way of token-mixing is reformed. With this design, ActiveMLP is endowed with the merits of global receptive fields and more flexible contentadaptive information fusion. Extensive experiments demonstrate that ActiveMLP is generally applicable and comprehensively surpasses different families of SOTA vision backbones by a clear margin on a broad range of vision tasks, including visual recognition and dense prediction tasks. The code and models will be available at https://github.com/microsoft/ActiveMLP.

Are we ready for a new paradigm shift? A survey on visual deep MLP

“…ActiveMLP 85 dynamically estimates the offset, rather than manually setting it like AS-MLP and CycleMLP do ( Figure 6 G). It first predicts the spatial locations of helpful contextual features along each direction at the channel level, and then finds and fuses them.…”

“…The approach based on shifting the feature map and channel projection further reduce computational complexity, i.e., the number of parameters is with FLOPs. MS-MLP 87 adds some depthwise convolutions, ActiveMLP 85 adds some channel projections, but both do not affect the overall complexity. Moreover, the number of weights is decoupled from the image resolution to no longer constrain these variants.…”

“…Object detection and semantic segmentation Some MLP-like variants 76,77,80,83,[85][86][87] pre-trained on ImageNet are transferred to downstream tasks, such as object detection and semantic segmentation. Such tasks are more challenging than classification due to involving multiple objects of interest in one input image.…”

Token-Label Alignment for Vision Transformers

“…ViTs can achieve better accuracy/computation trade-off than conventional CNNs, where one of the working mechanisms is the alternation between spatial mixing (e.g., SA) and channel mixing (e.g., MLP) (Tolstikhin et al, 2021). Based on this, some works have explored different spatial mixing strategies in addition to self-attention, including spatial MLP (Tolstikhin et al, 2021;Tang et al, 2022;Wei et al, 2022) and depth-wise convolution (Ding et al, 2022;Guo et al, 2022). For an image X ∈ R H×W ×C , they first perform patch-wise image tokenization to obtain a tokenized image representation Z ∈ R N ×d , where N is the number of tokens and d is the number of channels.…”

“…The recent developments of vision transformers (ViTs) have revolutionized the computer vision field and set new state-of-the-arts in a variety of tasks, such as image classification (Dosovitskiy et al, 2020;Chu et al, 2021), object detection (Carion et al, 2020;Zhu et al, 2020;Dai et al, 2021a;b), and semantic segmentation (Li et al, 2017;Strudel et al, 2021;Zheng et al, 2021;Cheng et al, 2021). The successful structure of alternative spatial mixing and channel mixing in ViTs also motivates the arising of high-performance MLP-like deep architectures (Tolstikhin et al, 2021;Tang et al, 2022;Wei et al, 2022) and promotes the evolution of better CNNs (Ding et al, 2022;Guo et al, 2022). In addition to architecture designs, an improved training strategy can also greatly boost the performance of a trained deep model (Jiang et al, 2021;Touvron et al, 2022;.…”