Tensor Low-Rank Reconstruction for Semantic Segmentation

Abstract

Context information plays an indispensable role in the success of semantic segmentation. Recently, non-local self-attention based methods are proved to be effective for context information collection. Since the desired context consists of spatial-wise and channel-wise attentions, 3D representation is an appropriate formulation. However, these non-local methods describe 3D context information based on a 2D similarity matrix, where space compression may lead to channel-wise attention missing. An alternative is to model the contextual information directly without compression. However, this effort confronts a fundamental difficulty, namely the high-rank property of context information. In this paper, we propose a new approach to model the 3D context representations, which not only avoids the space compression but also tackles the high-rank difficulty. Here, inspired by tensor canonical-polyadic decomposition theory (i.e, a high-rank tensor can be expressed as a combination of rank-1 tensors.), we design a low-rank-to-high-rank context reconstruction framework (i.e, RecoNet). Specifically, we first introduce the tensor generation module (TGM), which generates a number of rank-1 tensors to capture fragments of context feature. Then we use these rank-1 tensors to recover the high-rank context features through our proposed tensor reconstruction module (TRM). Extensive experiments show that our method achieves state-of-the-art on various public datasets. Additionally, our proposed method has more than 100 times less computational cost compared with conventional non-local-based methods.

STransFuse: Fusing Swin Transformer and Convolutional Neural Network for Remote Sensing Image Semantic Segmentation

“…The Feature Pyramid Transformer (FPT) [11] is a completely active feature interaction that extends the receptive field through the specified Transformer. Chen et al [24] developed a tensor generation module to capture contextual data and offered a new way to modeling 3D context representations. The spatial relation module and the channel relation module were introduced by Mou et al [25] to learn and infer the global link between any two spatial positions or feature maps, and then build a feature representation with improved relationship.…”

“…We used the ground truth whose boundaries of objects have not been eroded by 3-pixel radius for testing. According to the official division principle, 17 patches were used as the test set (image id: 1, 3,5,7,11,13,15,17,20,21,23,26,28,30,32,34,37), and the other 16 as the training set (image id: 2, 4, 6, 8, 10, 12,14,16,20,22,24,27,29,31,33,35,38). For the large image, we cut it into 256×256 slices.…”

Hyperspectral Classification Using Cooperative Spatial-Spectral Attention Network With Tensor Low-Rank Reconstruction

“…However, high-rank tensor representation needs a huge cost. Inspired by [19], we design a TLRR to deal with the above problem. This block consists of two 1 × 1 Conv layers, a low-rank tensor generation module (TGM) and a high-rank tensor reconstruction module (TRM), as shown in Fig.…”

“…It can be observed that the spatial-spectral information is seldom considered simultaneously during their attention features extraction phase in these methods. Inspired by [19], we propose a novel cooperative spatial-spectral attention network using tensor low-rank reconstruction, which is able to model a 3D attention map tensor considering the long-range dependencies of spatial land covers and spectral bands. In addition, spatial-spectral attention features are reconstructed to assist the network in enhancing the discriminative spatialspectral features.…”

Fully Attentional Network for Semantic Segmentation

“…These visualization results further demonstrate that our proposed module can capture and encode the spatial similarities into the channel attention map to achieve full attentions. (Zhang et al 2018) ResNet-101 85.9 DFN (Yu et al 2018) ResNet-101 86.2 CFNet (Zhang, Wang, and Xie 2019) ResNet-101 87.2 EMANet ResNet-101 87.7 DeeplabV3+ (Chen et al 2018) Xception+JFT 89.0 RecoNet (Chen et al 2020) ResNet-101 88.5 FLANet (Ours) ‡…”