MorphMLP: An Efficient MLP-Like Backbone for Spatial-Temporal Representation Learning

Abstract

Self-attention has become an integral component of the recent network architectures, e.g., Transformer, that dominate major image and video benchmarks. This is because self-attention can flexibly model long-range information. For the same reason, researchers make attempts recently to revive Multiple Layer Perceptron (MLP) and propose a few MLP-Like architectures, showing great potential. However, the current MLP-Like architectures are not good at capturing local details and lack progressive understanding of core details in the images and/or videos. To overcome this issue, we propose a novel MorphMLP architecture that focuses on capturing local details at the low-level layers, while gradually changing to focus on long-term modeling at the highlevel layers. Specifically, we design a Fully-Connected-Like layer, dubbed as MorphFC, of two morphable filters that gradually grow its receptive field along the height and width dimension. More interestingly, we propose to flexibly adapt our MorphFC layer in the video domain. To our best knowledge, we are the first to create a MLP-Like backbone for learning video representation. Finally, we conduct extensive experiments on image classification, semantic segmentation and video classification. Our MorphMLP, such a self-attention free backbone, can be as powerful as and even outperform self-attention based models. * David J. Zhang and K. Li contribute equally. † Work is done during internship at Meitu, Inc.

Video-Text Pre-training with Learned Regions

“…Video Representation Learning. In the past decade, a large number of deep 2D [18,27,39,63,71] and 3D [8,17,21,68,72,77] models have been proposed to extract efficient spatial and temporal representations for video. Recently, inspired by the success of Transformer in NLP field [14,69], visual Transformers [5,12,42,49] are sprung up for video representation.…”

Active Token Mixer

“…Afterwards, Transformers spring up and make splendid breakthroughs on various vision tasks [13,14,15,16,17,18,19,20,21,22,23,24,25,26,27]. Most recently, the multi-layer perceptrons (MLPs) based architectures [28,29] have regained their light and been demonstrated capable of achieving stunning results on vision tasks [30,28,31,32,29,33,34]. A situation in which these three families of backbone architectures are contending has been formed.…”

“…Transformer-based architectures [12,13,14,35,16,15,36,37,17,38,39] perform message passing from other tokens into the query token based on the calculated pairwise attention weights, depending on the affinities between tokens in the embedding space. MLPbased architectures mostly enable information interaction through spatial fully connections across all tokens [28,30,40,34] or across certain tokens selected with hand-crafted rules in a deterministic manner [31,33,41,32,29,42]. However, the fully connection across all tokens makes the network incapable of coping with variable input resolutions, limiting the usage on downstream tasks (e.g., object detection and segmentation).…”

“…However, the fully connection across all tokens makes the network incapable of coping with variable input resolutions, limiting the usage on downstream tasks (e.g., object detection and segmentation). Manually designing deterministic rules to restrict the spatial information interaction within fixed regions [31,33,32,29,41,42] eliminates this resolution-fixed constraint but suffers from the lack of adaptability to various visual contents with diverse feature patterns in information interaction.…”

Suppressing Static Visual Cues via Normalizing Flows for Self-Supervised Video Representation Learning

“…Recent top-performing approaches to solving video understanding tasks are based on supervised learning with a large amount of labeled data for training. Due to the strong data fitting capacity of deep convolutional neural networks, competitive performance can be achieved for recognizing actions in videos (Carreira et al 2017;Zhang et al 2021). One of the key factors for the success may owe to the strong correlation between action class and object/background known as representation bias in (Li et al 2018;Choi et al 2019).…”