Lifting Scheme-Based Deep Neural Network for Remote Sensing Scene Classification

Abstract

Recently, convolutional neural networks (CNNs) achieve impressive results on remote sensing scene classification, which is a fundamental problem for scene semantic understanding. However, convolution, the most essential operation in CNNs, restricts the development of CNN-based methods for scene classification. Convolution is not efficient enough for high-resolution remote sensing images and limited in extracting discriminative features due to its linearity. Thus, there has been growing interest in improving the convolutional layer. The hardware implementation of the JPEG2000 standard relies on the lifting scheme to perform wavelet transform (WT). Compared with the convolution-based two-channel filter bank method of WT, the lifting scheme is faster, taking up less storage and having the ability of nonlinear transformation. Therefore, the lifting scheme can be regarded as a better alternative implementation for convolution in vanilla CNNs. This paper introduces the lifting scheme into deep learning and addresses the problems that only fixed and finite wavelet bases can be replaced by the lifting scheme, and the parameters cannot be updated through backpropagation. This paper proves that any convolutional layer in vanilla CNNs can be substituted by an equivalent lifting scheme. A lifting scheme-based deep neural network (LSNet) is presented to promote network applications on computational-limited platforms and utilize the nonlinearity of the lifting scheme to enhance performance. LSNet is validated on the CIFAR-100 dataset and the overall accuracies increase by 2.48% and 1.38% in the 1D and 2D experiments respectively. Experimental results on the AID which is one of the newest remote sensing scene dataset demonstrate that 1D LSNet and 2D LSNet achieve 2.05% and 0.45% accuracy improvement compared with the vanilla CNNs respectively.

Lifting Scheme-Based Sparse Density Feature Extraction for Remote Sensing Target Detection

“…In our previous paper [23], we have proven the relationship between the lifting scheme and vanilla convolutional layer that…”

“…The lifting scheme implementation obtains exactly the same output but reduces the computation complexity compared with the traditional wavelet transform. Our previous paper [23] has applied the lifting scheme to substitute vanilla convolutional layers (with a stride of 1) in CNNs to enhance the accuracy of remote sensing scene classification. However, the relationship between the lifting scheme and the sparse density feature extraction has not been explored yet.…”

Simplified object-based deep neural network for very high resolution remote sensing image classification

“…CNNs can extract high-level features of remote sensing images and achieve accuracies much higher than those of traditional shallow models (Xia et al, 2017;Zhao et al, 2020). By modifying the structures or training mechanisms of traditional CNNs, more discriminative features of scenes can be discovered, and the accuracy can be further improved (He et al, 2019;Li et al, 2020a). CNNs can achieve very high accuracy in scene classification.…”

Lifting Based Object Detection Networks of Remote Sensing Imagery for FPGA Accelerator

“…2) To further explore the feasibility of the implementation, it is necessary to map the lifting scheme to the hardware structure. Based on the equivalence between the lifting scheme and the convolution [15], we introduced the lifting scheme into object detection, to promote the ability of hardware implementation while maintaining the same accuracy. The lifting based convolution layers provide a parallel strategy for any convolutional layer to obtain high throughput and high utilization efficiency of hardware resources.…”

“…1) Based on the equivalence between the lifting scheme and the convolution [15], we introduce the lifting scheme into YOLOv3-tiny [16] and propose a novel object detection network LS-YOLOv3-tiny, to enhance the hardware implementation ability of the networks while maintaining the same accuracy. 2) This paper proposed a lifting based FPGA accelerator.…”