Task-Driven Onboard Real-Time Panchromatic Multispectral Fusion Processing Approach for High-Resolution Optical Remote Sensing Satellite

Zhang, Z.; Lü, Wei; Shao, Xiang; Xie, Guangqi; Liu, Chuang; Xu, Ming

doi:10.1109/jstars.2023.3305231

Cited by 5 publications

(5 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Real-time performance is a key metric for UAV safe emergency landing area recognition applications, especially important on performance-constrained embedded devices [40][41][42][43]. In this study, inference speed is used as a metric for real-time performance evaluation.…”

Section: Resultsmentioning

confidence: 99%

“…where N denotes the number of semantic categories, TP i is the number of correctly categorized pixels, FP i is the number of pixels in the predicted image in which category i is categorized as other categories, FN i is the number of pixels in the predicted image in which other categories are categorized as category i, and TN i is the number of correctly categorized pixels in the predicted image in which other categories are correctly categorized except for category i. P and R stand for Precision and Recall, respectively. Real-time performance is a key metric for UAV safe emergency landing area recognition applications, especially important on performance-constrained embedded devices [40][41][42][43]. In this study, inference speed is used as a metric for real-time performance evaluation.…”

Section: Implementation Details and Metricsmentioning

confidence: 99%

See 1 more Smart Citation

KDP-Net: An Efficient Semantic Segmentation Network for Emergency Landing of Unmanned Aerial Vehicles

Zhang,

Xiang

et al. 2024

Drones

Self Cite

View full text Add to dashboard Cite

As the application of UAVs becomes more and more widespread, accidents such as accidental injuries to personnel, property damage, and loss and destruction of UAVs due to accidental UAV crashes also occur in daily use scenarios. To reduce the occurrence of such accidents, UAVs need to have the ability to autonomously choose a safe area to land in an accidental situation, and the key lies in realizing on-board real-time semantic segmentation processing. In this paper, we propose an efficient semantic segmentation method called KDP-Net for characteristics such as large feature scale changes and high real-time processing requirements during the emergency landing process. The proposed KDP module can effectively improve the accuracy and performance of the semantic segmentation backbone network; the proposed Bilateral Segmentation Network improves the extraction accuracy and processing speed of important feature categories in the training phase; and the proposed edge extraction module improves the classification accuracy of fine features. The experimental results on the UDD6 and SDD show that the processing speed of this method reaches 85.25 fps and 108.11 fps while the mIoU reaches 76.9% and 67.14%, respectively. The processing speed reaches 53.72 fps and 38.79 fps when measured on Jetson Orin, which can meet the requirements of airborne real-time segmentation for emergency landing.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Implementation Details and Metricsmentioning

confidence: 99%

KDP-Net: An Efficient Semantic Segmentation Network for Emergency Landing of Unmanned Aerial Vehicles

Zhang,

Xiang

et al. 2024

Drones

Self Cite

View full text Add to dashboard Cite

show abstract

“…The specific performance and power consumption of the Jeston AGX Orin is shown in Table 6 below: EDP represents no power consumption limit. In this paper, we use the method in Zhang's article [4] for the on-board ROI (Region of Interest) preprocessing of the data, where the ROI is 4000 pixel for the panchromatic color, and then the panchromatic and multispectral data are acquired in memory for fusion processing, respectively. This process is repeated three times, and the fusion time is averaged.…”

Section: Embedded Device Performance Testingmentioning

confidence: 99%

“…Facing future on-board processing requirements, constrained by the limitations of on-board processing on the satellite and the requirements of realtime, the newly designed algorithms should be characterized by a small computational volume, not relying on external memory, and suitable for block independent processing. Furthermore, due to the constraints imposed by on-board satellite processing capabilities and the need for real-time operations, it is crucial to develop algorithms with a reduced computational workload, independent processing of data blocks, and minimal reliance on external memory [4]. For the on-board multiscale remote sensing image fusion problem, the deep learning approach has higher computational complexity and greater computational power requirements, while the traditional approach simply adds the detail information of the panchromatic image to the multispectral image without considering the difference in spatial resolution of the two images, which has lower hardware requirements and is easier to achieve real-time processing.…”

Section: Introductionmentioning

confidence: 99%

Multiscale Fusion of Panchromatic and Multispectral Images Based on Adaptive Iterative Filtering

Zhang,

Xu,

Wang

et al. 2023

Remote Sensing

Self Cite

View full text Add to dashboard Cite

This paper proposes an efficient and high-fidelity image fusion method based on adaptive smoothing filtering for panchromatic (PAN) and multispectral (MS) image fusion. The scale ratio reflects the ratio of spatial resolution between the panchromatic image and the multispectral image. When facing a multiscale fusion task, traditional methods are unable to simultaneously handle the problems of spectral resolution loss resulting from high scale ratios and the issue of reduced spatial resolution due to low scale ratios. To adapt to the fusion of panchromatic and multispectral satellite images of different scales, this paper improves the problem of the insufficient filtering of high-frequency information of remote sensing images of different scales by the classic filter-based intensity modulation (SFIM) model. It uses Gaussian convolution kernels instead of traditional mean convolution kernels and builds a Gaussian pyramid to adaptively construct convolution kernels of different scales to filter out high-frequency information of high-resolution images. It can adaptively process panchromatic multispectral images of different scales, iteratively filter the spatial information in panchromatic images, and ensure that the scale transformation is consistent with the definition of multispectral images. Using 15 common fusion methods, this paper compares the experimental results of ZY-3 with scale ratio 2.7 and SV-1 with scale ratio 4 data. The results show that the method proposed in this paper retains good spatial information for image fusion at different scales and has good spectral preservation.

show abstract

“…Furthermore, in practical scenarios, having fewer parameters can reduce the device specification requirement for the fusion method. This, in turn, provides a more flexible margin within the same specifications for subsequent tasks [14]. On the other hand, it is capable of dividing each computational unit within the embedded device into larger units, leading to enhanced performance in constrained application scenarios, such as on-board processing [15][16][17][18][19].…”

Section: Introductionmentioning

confidence: 99%

Recursive Self-Attention Modules-Based Network for Panchromatic and Multispectral Image Fusion

Liu,

Wei,

Zhang

et al. 2023

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

Self Cite

View full text Add to dashboard Cite

In the field of remote sensing, image fusion technology plays a crucial role in observing the state of global resources and environmental conditions, proposing response strategies, and constantly monitoring and correcting strategies. Currently, the majority of traditional methods exhibit varying degrees of spatial or spectral distortion, and these unreasonable spectral distributions may contain erroneous geographical feature information. Meanwhile, despite their performance in fusion results, deep learning-based methods cannot be applied to some practical application scenarios due to the requirement for hardware specifications resulting from the large number of parameters in their models. These issues are not conducive to accurately reflecting the actual geomorphic resource conditions or promoting sustainable development. In order to address the above issues, we propose a novel recursive self-attention module (RSAM), which consists of two stages: spatial-spectral similarity extraction and self-attention weight generation. The proposed RSAM employs a G2L strategy to capture the global interdependencies of two distinct local locations in the feature map. This method allows for simultaneous consideration of both spatial and spectral information, while focusing on more mutual information between spectral and spatial dimensions. Subsequently, we construct the corresponding residual block (RSARB) through RSAM and concatenate the RSARBs to generate RSANet with a limited number of parameters. Extensive experiments demonstrate that RSANet achieves superior results in both qualitative and quantitative evaluation, despite the model parameters being within a narrow range of orders of magnitude. This demonstrates that the proposed method possesses robust feature learning capability and practicality for observing and studying the global resource environment. The source code will be publicly available at https://github.com/JUSTM0VE0N/RSANet.

show abstract

Task-Driven Onboard Real-Time Panchromatic Multispectral Fusion Processing Approach for High-Resolution Optical Remote Sensing Satellite

Cited by 5 publications

References 36 publications

KDP-Net: An Efficient Semantic Segmentation Network for Emergency Landing of Unmanned Aerial Vehicles

KDP-Net: An Efficient Semantic Segmentation Network for Emergency Landing of Unmanned Aerial Vehicles

Multiscale Fusion of Panchromatic and Multispectral Images Based on Adaptive Iterative Filtering

Recursive Self-Attention Modules-Based Network for Panchromatic and Multispectral Image Fusion

Contact Info

Product

Resources

About