Rational Polynomial Camera Model Warping for Deep Learning Based Satellite Multi-View Stereo Matching

Gao, Jian; Liu, Jin; Ji, Shunping

doi:10.1109/iccv48922.2021.00609

Cited by 18 publications

(10 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The main advantage of traditional pipelines is preserved as no explicit geometry supervision is required: the learning is self-supervised and based solely on the color of the input images. This is a key difference with respect to other state-of-the-art deep learning methods for DSM generation from satellite imagery [7,17,18,48], which depend on ground truth geometry models.…”

Section: Related Workmentioning

confidence: 98%

Sat-NeRF: Learning Multi-View Satellite Photogrammetry With Transient Objects and Shadow Modeling Using RPC Cameras

Marí¹,

Facciolo²,

Ehret³

2022

Preprint

View full text Add to dashboard Cite

We introduce the Satellite Neural Radiance Field (Sat-NeRF), a new end-to-end model for learning multi-view satellite photogrammetry in the wild. Sat-NeRF combines some of the latest trends in neural rendering with native satellite camera models, represented by rational polynomial coefficient (RPC) functions. The proposed method renders new views and infers surface models of similar quality to those obtained with traditional state-of-the-art stereo pipelines. Multi-date images exhibit significant changes in appearance, mainly due to varying shadows and transient objects (cars, vegetation). Robustness to these challenges is achieved by a shadow-aware irradiance model and uncertainty weighting to deal with transient phenomena that cannot be explained by the position of the sun. We evaluate Sat-NeRF using WorldView-3 images from different locations and stress the advantages of applying a bundle adjustment to the satellite camera models prior to training. This boosts the network performance and can optionally be used to extract additional cues for depth supervision.

show abstract

Section: Related Workmentioning

confidence: 98%

Sat-NeRF: Learning Multi-View Satellite Photogrammetry With Transient Objects and Shadow Modeling Using RPC Cameras

Marí¹,

Facciolo²,

Ehret³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The rational polynomial camera model (RPC) is extensively used in satellite imagery processing, which connects the image points and corresponding world coordinate points with cubic rational polynomial coefficients (Gao et al, 2021). We define the world coordinates as (lat n , lon n , hei n ) which represents the latitude, longitude and height.…”

Section: Rational Polynomial Camera Model (Rpc)mentioning

confidence: 99%

“…Our A-SATMVSNet is implemented using PyTorch, which is trained on TLC SatMVS training dataset for evaluation on TLC SatMVS testing dataset. The preprocessing strategies and selection of input views follow common strategies in a representative previous work (Gao et al, 2021). We train and validate our model on the TLC SatMVS training set and evaluating set respectively.…”

Section: Trainingmentioning

confidence: 99%

A-SATMVSNet: An attention-aware multi-view stereo matching network based on satellite imagery

et al. 2023

View full text Add to dashboard Cite

Introduction: The stereo matching technology of satellite imagery is an important way to reconstruct real world. Most stereo matching technologies for satellite imagery are based on depth learning. However, the existing depth learning based methods have the problems of holes and matching errors in stereo matching tasks.Methods: In order to improve the effect of satellite image stereo matching results, we propose a satellite image stereo matching network based on attention mechanism (A-SATMVSNet). To solve the problem of insufficient extraction of surface features, a new feature extraction module based on triple dilated convolution with attention module is proposed, which solves the problem of matching holes caused by insufficient extraction of surface features. At the same time, compared with the traditional weighted average method, we design a novel cost-volume method that integrates attention mechanism to reduce the impact of matching errors to improve the accuracy of matching.Results and discussion: Experiments on public multi-view stereo matching dataset based on satellite imagery demonstrate that the proposed method significantly improves the accuracy and outperforms various previous methods. Our source code is available at https://github.com/MVSer/A-SATMVSNet.

show abstract

“…The Digital Surface Model (DSM), as one of the most fundamental geographical products, has been widely used in various applications, such as 3D city modelling, 3D land‐use management and disaster monitoring (Gruen et al., 2013; Han et al., 2020; Liu et al., 2023; Lv et al., 2022; Lv, Zhong, Wang, You, & Falco, 2023; Lv, Zhong, Wang, You, & Shi, 2023; Zhao et al., 2022). Satellite stereos, due to their characteristics of flexible acquisition and low cost, have been the dominant data source for generating city‐ or country‐level DSMs (Bosch et al., 2016; Gao et al., 2021; Huang et al., 2016; Kendall et al., 2017; Leotta et al., 2019; Lv, Zhong, Wang, You, & Falco, 2023; Zhang, Cui, et al., 2022). Thus, satellite image stereo matching (SISM) continues to be a hot research topic in recent years (Huang & Qin, 2020; Michel et al., 2020; Qin, 2019a; Zhang et al., 2017, 2019).…”

Section: Introductionmentioning

confidence: 99%

“…However, limited by the insufficient description capability, conventional algorithms always suffer from serious mismatching problems in intractable regions, including texture‐less and repetitive regions (i.e., farmland), heavily occluded regions (i.e., dense buildings) and other terrains (Facciolo et al., 2017; Huang et al., 2018; Qin, 2019b). Recent works show that the mismatching rates of stereo matching can be significantly decreased by using deep‐learning technical as solvers (Gao et al., 2021; Ji et al., 2019; Shen et al., 2020). However, though deep learning‐based stereo matching methods (DLSMs) have flourished in recent years, applying DLSMs on high‐resolution satellite stereos with broad image coverage and wide terrain variety is still challenging (Bosch et al., 2016; Chang & Chen, 2018; Shen et al., 2021; Xu et al., 2022).…”

Section: Introductionmentioning

confidence: 99%

Digital surface model generation from high‐resolution satellite stereos based on hybrid feature fusion network

Zheng,

Wan,

Zhang

et al. 2024

The Photogrammetric Record

View full text Add to dashboard Cite

Recent studies have demonstrated that deep learning‐based stereo matching methods (DLSMs) can far exceed conventional ones on most benchmark datasets by both improving visual performance and decreasing the mismatching rate. However, applying DLSMs on high‐resolution satellite stereos with broad image coverage and wide terrain variety is still challenging. First, the broad coverage of satellite stereos brings a wide disparity range, while DLSMs are limited to a narrow disparity range in most cases, resulting in incorrect disparity estimation in areas with contradictory disparity ranges. Second, high‐resolution satellite stereos always comprise various terrain types, which is more complicated than carefully prepared datasets. Thus, the performance of DLSMs on satellite stereos is unstable, especially for intractable regions such as texture‐less and occluded regions. Third, generating DSMs requires occlusion‐aware disparity maps, while traditional occlusion detection methods are not always applicable for DLSMs with continuous disparity. To tackle these problems, this paper proposes a novel DLSM‐based DSM generation workflow. The workflow comprises three steps: pre‐processing, disparity estimation and post‐processing. The pre‐processing step introduces low‐resolution terrain to shift unmatched disparity ranges into a fixed scope and crops satellite stereos to regular patches. The disparity estimation step proposes a hybrid feature fusion network (HF2Net) to improve the matching performance. In detail, HF2Net designs a cross‐scale feature extractor (CSF) and a multi‐scale cost filter. The feature extractor differentiates structural‐context features in complex scenes and thus enhances HF2Net's robustness to satellite stereos, especially on intractable regions. The cost filter filters out most matching errors to ensure accurate disparity estimation. The post‐processing step generates initial DSM patches with estimated disparity maps and then refines them for the final large‐scale DSMs. Primary experiments on the public US3D dataset showed better accuracy than state‐of‐the‐art methods, indicating HF2Net's superiority. We then created a self‐made Gaofen‐7 dataset to train HF2Net and conducted DSM generation experiments on two Gaofen‐7 stereos to further demonstrate the effectiveness and practical capability of the proposed workflow.

show abstract

Rational Polynomial Camera Model Warping for Deep Learning Based Satellite Multi-View Stereo Matching

Cited by 18 publications

References 26 publications

Sat-NeRF: Learning Multi-View Satellite Photogrammetry With Transient Objects and Shadow Modeling Using RPC Cameras

Sat-NeRF: Learning Multi-View Satellite Photogrammetry With Transient Objects and Shadow Modeling Using RPC Cameras

A-SATMVSNet: An attention-aware multi-view stereo matching network based on satellite imagery

Digital surface model generation from high‐resolution satellite stereos based on hybrid feature fusion network

Contact Info

Product

Resources

About