FusionNet: A Convolution–Transformer Fusion Network for Hyperspectral Image Classification

Yang, Liming; Yang, Yihang; Yang, Jianguo; Zhao, Ningyuan; Wu, Ling; Wang, Liguo; Wang, Tianrui

doi:10.3390/rs14164066

Cited by 39 publications

(21 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Niu et al propose a transformer based semantic segmentation model 21 for crop mapping task and prove to be effective. Yang et al proposed a convolutiontransformer fusion network for HSI classification, 22 which fuses the convolution and transformer in both serial and parallel mechanisms to achieve the full utilization of HSI features. Hu et al proposed a transformer-based fusion network for HSI super-resolution, which achieves efficient results.…”

Section: Transformermentioning

confidence: 99%

See 1 more Smart Citation

HFC-SST: improved spatial-spectral transformer for hyperspectral few-shot classification

Huang

Tang

et al. 2023

J. Appl. Rem. Sens.

View full text Add to dashboard Cite

Owing to the complex environment of hyperspectral image (HSI) collecting area, it is difficult to obtain an extensive number of labeled samples for HSI. Recently, many few-shot learning (FSL) algorithms based on convolutional neural network (CNN) have been employed for HSI classification in the scenery of small-scale training samples. However, a CNN-based model is unsuitable for modeling the spatial-spectral information with long-range dependency. The transformer has proved its superiority in modeling the long-range dependency. Inspired by this, an improved spatial-spectral transformer for HSI few-shot classification (HFC-SST) is proposed to deeply extract the local spatial-spectral information with only a few labeled samples. The contribution of this letter is twofold. First, a local spatial-spectral sequence generation method based on spatial-spectral correlation analysis and adjacent position information is proposed to generate the input sequence for transformer. Second, a local spatial-spectral feature extraction network based on the transformer is proposed to further exploit the spatial-spectral feature information on the input sequence. Experimental results on HFC with four datasets confirm that our proposed HFC-SST algorithm can achieve higher classification accuracy than the traditional CNN algorithms and the HSI FSL algorithms.

show abstract

Section: Transformermentioning

confidence: 99%

“…Yang et al. proposed a convolution-transformer fusion network for HSI classification, 22 which fuses the convolution and transformer in both serial and parallel mechanisms to achieve the full utilization of HSI features. Hu et al.…”

Section: Related Workmentioning

confidence: 99%

HFC-SST: improved spatial-spectral transformer for hyperspectral few-shot classification

Huang

Tang

et al. 2023

J. Appl. Rem. Sens.

View full text Add to dashboard Cite

show abstract

“…The works [50], [51], [52], [53] classify HSIs using a spectral-spatial approach, using steps to extract spatial information and using the spectral signature, and applying the concept of Vision Transformer (ViT). There are authors, as [54], [55], [56], that unite features of CNN and LSTM architecture with Transformer for HSI classification. Some works like [57], [58] also present an extensive comparison of the transformer architecture with other types of architectures, comparing approaches using spectral and spatial information.…”

Section: Introductionmentioning

confidence: 99%

Hyperspectral Image Classification: An Analysis Employing CNN, LSTM, Transformer, and Attention Mechanism

et al. 2023

View full text Add to dashboard Cite

Hyperspectral images contain tens to hundreds of bands, implying a high spectral resolution. This high spectral resolution allows for obtaining a precise signature of structures and compounds that make up the captured scene. Among the types of processing that may be applied to Hyperspectral Images, classification using machine learning models stands out. The classification process is one of the most relevant steps for this type of image. It can extract information using spatial and spectral information and spatial-spectral fusion. Artificial Neural Network models have been gaining prominence among existing classification techniques. They can be applied to data with one, two, or three dimensions. Given the above, this work evaluates Convolutional Neural Network models with one, two, and three dimensions to identify the impact of classifying Hyperspectral Images with different types of convolution. We also expand the comparison to Recurrent Neural Network models, Attention Mechanism, and the Transformer architecture.. Furthermore, a novelty pre-processing method is proposed for the classification process to avoid generating data leaks between training, validation, and testing data. The results demonstrated that using 1 Dimension Convolutional Neural Network (1D-CNN), Long Short-Term Memory (LSTM), and Transformer architectures reduces memory consumption and sample processing time and maintain a satisfactory classification performance up to 99% accuracy on larger datasets. In addition, the Transfomer architecture can approach the 2D-CNN and 3D-CNN architectures in accuracy using only spectral information. The results also show that using two or three dimensions convolution layers improves accuracy at the cost of greater memory consumption and processing time per sample. Furthermore, the pre-processing methodology guarantees the disassociation of training and testing data.

show abstract

“…Wang [22] proposed a lightweight MILRDA model, which extracts more discriminative features by constructing attention blocks while retaining shallow features, and suppresses useless background by channel attention, and achieves good results on UCM Datasets and other data sets. Yang [23] proposed a fusion network FusionNet for HSI classification and transformation, which integrates convolution and transformation operations in serial and parallel mechanisms, and the experimental results on small-scale data sets are promising. Bai [24] proposed an adaptive dual attention network, in which decibels processed the features from both spectral and spatial angles, strengthening the independent feature representation of the spectrum, improved the ability to search for features, and then aggregated high-level features by evaluating the confidence of effective information in the receptive field, designed the dispersion loss, supervised the learnable parameters, and improved the object recognition performance, making good progress on publicly available datasets.…”

Section: Introductionmentioning

confidence: 99%

Research on identification and classification of grassland forage based on deep learning and attention mechanisms

Liu

Zhao

et al. 2023

IET Image Processing

View full text Add to dashboard Cite

Grassland is an important resource for China's economic development and the main economic source of animal husbandry. The identification and classification of grassland forage is an important part of the improvement of forage varieties and the monitoring of germplasm resources, which can fundamentally solve the problems of poor forage quality and low reproduction rate. For the problem of low accuracy of forage identification and classification, the authors put forward a new 3DSECNN model to remove the preprocessing operation and directly study the images. The authors took forage hyperspectral image (HSI) images on the field and built dataset, used 3DSECNN to train the images to improve the classification effect. The outstanding contributions of this paper are: (1) The authors took high‐precision forage HSI images in the field, established a dedicated database of forage HSIs, and expanded the datasets; (2) the process of integrating preprocessing ideas into the network and replacing the traditional method of preprocessing the data and then extracting features; (3) proposing the 3DSECNN model, which adds SENet on the basis of the traditional 3DCNN, strengthens the correlation of the spatial dimension, selects the key features for the classification by calculating the channel weight, inhibits the unimportant information, and achieves the purpose of integrating the preprocessing ideas into the network. The experimental results show the overall accuracy (OA) of 3DSECNN is 94.36%, Precision, Recall, F1‐score, Kappa, and Time also showed good levels. The experimental results prove that the 3DSECNN strengthens the correlation between image channels, enhances the performance ability of features, and provides a new method for the identification and classification research of forage.

show abstract

FusionNet: A Convolution–Transformer Fusion Network for Hyperspectral Image Classification

Cited by 39 publications

References 49 publications

HFC-SST: improved spatial-spectral transformer for hyperspectral few-shot classification

HFC-SST: improved spatial-spectral transformer for hyperspectral few-shot classification

Hyperspectral Image Classification: An Analysis Employing CNN, LSTM, Transformer, and Attention Mechanism

Research on identification and classification of grassland forage based on deep learning and attention mechanisms

Contact Info

Product

Resources

About