3D Reconstruction of Remote Sensing Mountain Areas with TSDF-Based Neural Networks

Qi, Zhongyang; Zou, Zhengxia; Chen, Hao; Shi, Zhenwei

doi:10.3390/rs14174333

Cited by 9 publications

(8 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Implicit neural representation is an emerging technique for characterizing continuous signals based on neural networks. In computer graphics and computer vision, implicit neural representations were first used to represent 3D scenes [14][15][16][17]. To optimize the neural network, these methods usually need 3D data for supervision.…”

Section: B Implicit Neural Segmentationmentioning

confidence: 99%

Implicit Ray-Transformers for Multi-view Remote Sensing Image Segmentation

Qi¹,

Chen²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

The mainstream CNN-based remote sensing (RS) image semantic segmentation approaches typically rely on massive labeled training data. Such a paradigm struggles with the problem of RS multi-view scene segmentation with limited labeled views due to the lack of considering 3D information within the scene. In this paper, we propose "Implicit Ray-Transformer (IRT)" based on Implicit Neural Representation (INR), for RS scene semantic segmentation with sparse labels (such as 4-6 labels per 100 images). We explore a new way of introducing multiview 3D structure priors to the task for accurate and viewconsistent semantic segmentation. The proposed method includes a two-stage learning process. In the first stage, we optimize a neural field to encode the color and 3D structure of the remote sensing scene based on multi-view images. In the second stage, we design a Ray Transformer to leverage the relations between the neural field 3D features and 2D texture features for learning better semantic representations. Different from previous methods that only consider 3D prior or 2D features, we incorporate additional 2D texture information and 3D prior by broadcasting CNN features to different point features along the sampled ray. To verify the effectiveness of the proposed method, we construct a challenging dataset containing six synthetic subdatasets collected from the Carla platform and three real subdatasets from Google Maps. Experiments show that the proposed method outperforms the CNN-based methods and the state-ofthe-art INR-based segmentation methods in quantitative and qualitative metrics. Ablation study shows that under limited label conditions, the combination of the 3D structure prior and 2D texture can significantly improve the performance and effectively complete missing semantic information in novel views. Experiments also demonstrate the proposed method could yield geometry-consistent segmentation results against illumination changes and viewpoint changes. Our data and code will be public.

show abstract

Section: B Implicit Neural Segmentationmentioning

confidence: 99%

Implicit Ray-Transformers for Multi-view Remote Sensing Image Segmentation

Qi¹,

Chen²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…IMAGE fusion [1], 3D reconstruction [2], and target recognition [3]. Nowadays, research generally focuses on high-quality remote-sensing images with relatively simple background information.…”

Section: Introductionmentioning

confidence: 99%

A Heterogeneous Remote Sensing Image Matching Method for Urban Areas With Complex Terrain Based on 3D Spatial Relationship Constraints

Zheng,

Yang,

et al. 2024

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

Heterogeneous high-resolution remote sensing image matching will be disturbed by the differences in sensor type, imaging angle, height, and imaging time, and the matching difficulty is further increased in complex scenes with dense urban buildings and noticeable height differences. This paper proposes a method for matching heterogeneous high-resolution remote sensing images based on partitioned feature extraction and threedimensional spatial constraints. First, this paper conducts image partitioning based on the geometric differences of ground objects. Two feature extraction methods, namely, adaptive phase threshold and weighted moment map, are employed to extract feature points independently. To address the issue of inaccurate feature descriptions caused by drastic changes in viewing angles in buildings, we construct a robust feature descriptor by combining a multi-scale phase weighted energy convolution histogram (MSPW-ECH) with a new gradient location orientation histogram (GLOH)-like local feature descriptor. Additionally, a new similarity measure incorporating three-dimensional spatial constraints and the Marginalizing Sample Consensus (MAGSAC) method is applied to eliminate mismatched point pairs, ensuring the acquisition of precise matching points. Based on the feature detection results of two different synthetic data sets, it is evident that the proposed detector outperforms the three classical detectors in terms of repeatability and uniformity. Ultimately, the matching performance is experimentally verified on six groups of heterogeneous high-resolution remote sensing images. The experimental results show that the proposed method significantly outperforms RIFT, HAPCG, and MS-HLMO methods and achieves the best matching accuracy results.

show abstract

“…The development of 3D reconstruction has been a long process. Early 3D reconstruction was mainly completed by manual drawing, which was time-consuming and labor-intensive [4]. Nowadays, the main 3D reconstruction techniques can be divided into image-based 3D reconstruction and laser-scanner-based 3D reconstruction, which use different types of equipment (camera and laser scanner, respectively) to perform tasks [5].…”

Section: Introductionmentioning

confidence: 99%

Overview of Underwater 3D Reconstruction Technology Based on Optical Images

Wang

Shen

et al. 2023

JMSE

View full text Add to dashboard Cite

At present, 3D reconstruction technology is being gradually applied to underwater scenes and has become a hot research direction that is vital to human ocean exploration and development. Due to the rapid development of computer vision in recent years, optical image 3D reconstruction has become the mainstream method. Therefore, this paper focuses on optical image 3D reconstruction methods in the underwater environment. However, due to the wide application of sonar in underwater 3D reconstruction, this paper also introduces and summarizes the underwater 3D reconstruction based on acoustic image and optical–acoustic image fusion methods. First, this paper uses the Citespace software to visually analyze the existing literature of underwater images and intuitively analyze the hotspots and key research directions in this field. Second, the particularity of underwater environments compared with conventional systems is introduced. Two scientific problems are emphasized by engineering problems encountered in optical image reconstruction: underwater image degradation and the calibration of underwater cameras. Then, in the main part of this paper, we focus on the underwater 3D reconstruction methods based on optical images, acoustic images and optical–acoustic image fusion, reviewing the literature and classifying the existing solutions. Finally, potential advancements in this field in the future are considered.

show abstract

3D Reconstruction of Remote Sensing Mountain Areas with TSDF-Based Neural Networks

Cited by 9 publications

References 35 publications

Implicit Ray-Transformers for Multi-view Remote Sensing Image Segmentation

Implicit Ray-Transformers for Multi-view Remote Sensing Image Segmentation

A Heterogeneous Remote Sensing Image Matching Method for Urban Areas With Complex Terrain Based on 3D Spatial Relationship Constraints

Overview of Underwater 3D Reconstruction Technology Based on Optical Images

Contact Info

Product

Resources

About