FTC-Net: Fusion of Transformer and CNN Features for Infrared Small Target Detection

Qi, Meibin; Liu, Liu; Zhuang, Shuo; Liu, Yimin; Li, Kunyuan; Yang, Yue; Li, Xiaohong

doi:10.1109/jstars.2022.3210707

Cited by 34 publications

(11 citation statements)

References 42 publications

(52 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, some deep learning algorithms [36][37][38][39] are also used to detect small targets. Qi et al provided a fusion network of Transformer and a CNN (FTC-Net) [37], which extracts local detail features and global contextual features.…”

Section: The Deep Learning Methodsmentioning

confidence: 99%

A Space Target Detection Method Based on Spatial–Temporal Local Registration in Complicated Backgrounds

Su,

Chen,

Cang

et al. 2024

Remote Sensing

View full text Add to dashboard Cite

Human space exploration has brought a growing crowded operating environment for in-orbit spacecraft. Monitoring the space environment and detecting space targets with photoelectric equipment has extensive and realistic significance in space safety. In this study, a local spatial–temporal registration (LSTR) method is proposed to detect moving small targets in space. Firstly, we applied the local region registration to estimate the neighbor background motion model. Secondly, we analyzed the temporal local grayscale difference between the strong clutter and target region and measured the temporal local–central region difference to enhance the target. Then, the temporal pixel contrast map was calculated, which further retains the target signal and suppresses the residue clutter. Finally, a simple adaptive threshold segmentation algorithm was applied to the saliency map to segment the targets. Comparative experiments were conducted on four groups of image sequences to validate the efficiency and robustness of the algorithm. The experimental findings indicate that the proposed method performs well in target enhancement and clutter suppression under different scenarios.

show abstract

Section: The Deep Learning Methodsmentioning

confidence: 99%

A Space Target Detection Method Based on Spatial–Temporal Local Registration in Complicated Backgrounds

Su,

Chen,

Cang

et al. 2024

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…This makes CNN more suitable for extracting and encoding detailed features from low-level semantic feature layers [27][28].Therefore, in the design of the hybrid encoder for the MS-DETR model, we utilized a CNN structure in the feature extraction module to extract detailed information about weeds from the lowlevel semantic feature layer. When using the CNN network to extract low-level details, appropriately expanding the receptive eld of the CNN network enables it to capture richer features of the target and surrounding background areas, thereby improving the quality of small target detection [29][30]. Dilation convolution, compared to regular convolution, can enlarge the receptive eld, obtaining broader and richer features, which is crucial for detecting small targets of different scales [31][32].…”

Section: 1: Methodology Employed In This Studymentioning

confidence: 99%

Identifying Rice Field Weeds from Unmanned Aerial Vehicle Remote Sensing Imagery Using Deep Learning

Guo,

Cai,

Zhou

et al. 2024

Preprint

View full text Add to dashboard Cite

Background Rice field weed object detection can provide key information on weed species and locations for precise spraying, which is of great significance in actual agricultural production. However, facing the complex and changing real farm environments, traditional object detection methods still have difficulties in identifying small-sized, occluded and densely distributed weed instances. To address these problems, this paper proposes a multi-scale feature enhanced DETR network, named MS-DETR. By adding multi-scale feature extraction branches on top of DETR, this model fully utilizes the information from different semantic feature layers to improve recognition capability for rice field weeds in real-world scenarios. Methods Introducing multi-scale feature layers on the basis of the DETR model, we conduct a differentiated design for different semantic feature layers. The high-level semantic feature layer adopts Transformer structure to extract contextual information between barnyard grass and rice plants. The low-level semantic feature layer uses CNN structure to extract local detail features of barnyard grass. Introducing multi-scale feature layers inevitably leads to increased model computation, thus lowering model inference speed. Therefore, we employ a new type of Pconv (Partial convolution) to replace traditional standard convolutions in the model, so as to reduce memory access time and computational redundancy. Results On our constructed rice field weed dataset, compared with the original DETR model, our proposed MS-DETR model improved average recognition accuracy of rice field weeds by 2.8%, reaching 0.792. The MS-DETR model size is 40.8M with inference time of 0.0081 seconds. Compared with three classical DETR models (Deformable DETR, Anchor DETR and DAB-DETR), the MS-DETR model respectively improved average precision by 2.1%, 4.9% and 2.4%. Discussion This model has advantages such as high recognition accuracy and fast recognition speed. It is capable of accurately identifying rice field weeds in complex real-world scenarios, thus providing key technical support for precision spraying and management of variable-rate spraying systems.

show abstract

“…Then, they adopted ViT to learn high-level information of target localization from local features. Next, Qi et.al [39] proposed fusion network architecture of transformer and CNN (FTC-Net), which consists of two branches. The CNN-based branch uses a U-Net with skip connections to obtain low-level local details of small targets.…”

Section: Vision Transformermentioning

confidence: 99%

MTU-Net: Multilevel TransUNet for Space-Based Infrared Tiny Ship Detection

Luo

et al. 2023

IEEE Trans. Geosci. Remote Sensing

View full text Add to dashboard Cite

Space-based infrared tiny ship detection aims at separating tiny ships from the images captured by earth orbiting satellites. Due to the extremely large image coverage area (e.g., thousands square kilometers), candidate targets in these images are much smaller, dimer, more changeable than those targets observed by aerial-based and land-based imaging devices. Existing short imaging distance-based infrared datasets and target detection methods cannot be well adopted to the space-based surveillance task. To address these problems, we develop a spacebased infrared tiny ship detection dataset (namely, NUDT-SIRST-Sea) with 48 space-based infrared images and 17598 pixel-level tiny ship annotations. Each image covers about 10000 square kilometers of area with 10000 × 10000 pixels. Considering the extreme characteristics (e.g., small, dim, changeable) of those tiny ships in such challenging scenes, we propose a multi-level TransUNet (MTU-Net) in this paper. Specifically, we design a Vision Transformer (ViT) Convolutional Neural Network (CNN) hybrid encoder to extract multi-level features. Local feature maps are first extracted by several convolution layers and then fed into the multi-level feature extraction module (MVTM) to capture long-distance dependency. We further propose a copy-rotateresize-paste (CRRP) data augmentation approach to accelerate the training phase, which effectively alleviates the issue of sample imbalance between targets and background. Besides, we design a FocalIoU loss to achieve both target localization and shape description. Experimental results on the NUDT-SIRST-Sea dataset show that our MTU-Net outperforms traditional and existing deep learning based SIRST methods in terms of probability of detection, false alarm rate and intersection over union. Our code is available at: https://github.com/TianhaoWu16/Multi-level-TransUNet-for-Space-based-Infrared-Tiny-ship-Detection.

show abstract

FTC-Net: Fusion of Transformer and CNN Features for Infrared Small Target Detection

Cited by 34 publications

References 42 publications

A Space Target Detection Method Based on Spatial–Temporal Local Registration in Complicated Backgrounds

A Space Target Detection Method Based on Spatial–Temporal Local Registration in Complicated Backgrounds

Identifying Rice Field Weeds from Unmanned Aerial Vehicle Remote Sensing Imagery Using Deep Learning

MTU-Net: Multilevel TransUNet for Space-Based Infrared Tiny Ship Detection

Contact Info

Product

Resources

About