Pano3D: A Holistic Benchmark and a Solid Baseline for 360° Depth Estimation

Albanis, Georgios; Zioulis, Nikolaos; Drakoulis, Petros; Gkitsas, Vasileios; Sterzentsenko, Vladimiros; Álvarez, Federico; Zarpalas, Dimitrios; Daras, Petros

doi:10.1109/cvprw53098.2021.00413

Cited by 30 publications

(17 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For our analysis we use a dense regression task, namely depth estimation, which requires the balancing of both boundary preservation and smoothness of the predicted depth maps, apart from its direct depth estimation performance. To fully exploit rich depth maps that include both smooth regions, as well as lots of foreground to background depth discontinuities, we use an omnidirectional image benchmark [1]. It includes spherical panoramas that capture entire indoor scenes, containing a lot of flat surfaces (ceiling, floors, tables, etc.…”

Section: Resultsmentioning

confidence: 99%

“…Our implementation is based on moai [37] which uses PyTorch 1.8 [40], PyTorch Lightning 1.0.7 [11] and Kornia 0.4.1 [45]. For all experiments we use the same UNet architecture and supervision scheme used in Pano3D [1], fixing the learning rate (0.0002), optimizer (default parameterized Adam [25]), batch size (4) and random number generator seed. Thus, only the skip connection varies from experiment to experiment.…”

Section: Resultsmentioning

confidence: 99%

“…To perform an aggregated analysis among many metrics of different performance traits, namely direct depth, boundary preservation and smoothness, we use the following indicators derived from the metrics used in [1], which aggregate accuracy and error metrics:…”

Section: Hybrid Skip Connection Analysismentioning

confidence: 99%

See 2 more Smart Citations

Hybrid Skip: A Biologically Inspired Skip Connection for the UNet Architecture

et al. 2022

Self Cite

View full text Add to dashboard Cite

In this work we introduce a biologically inspired long-range skip connection for the UNet architecture that relies on the perceptual illusion of hybrid images, being images that simultaneously encode two images. The fusion of early encoder features with deeper decoder ones allows UNet models to produce finer-grained dense predictions. While proven in segmentation tasks, the network's benefits are downweighted for dense regression tasks as these long-range skip connections additionally result in texture transfer artifacts. Specifically for depth estimation, this hurts smoothness and introduces false positive edges which are detrimental to the task due to the depth maps' piece-wise smooth nature. The proposed HybridSkip connections show improved performance in balancing the trade-off between edge preservation, and the minimization of texture transfer artifacts that hurt smoothness. This is achieved by the proper and balanced exchange of information that HybridSkip connections offer between the high and low frequency, encoder and decoder features, respectively. The code and models will be made available in the project page.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Hybrid Skip: A Biologically Inspired Skip Connection for the UNet Architecture

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…Similar to the previous works, we estimate our method with the commonly used error metrics: MAE, MRE, RMSE, RMSE log scale invariant, and δ t , t ∈ 1.25, 1.25 2 , 1.25 3 . Specially, the metrics of dbe acc , dbe comp , prec t , and rec t , t ∈ 0.25, 0.5, 1 provided by Pano3D [43] are employed to evaluate the capability of boundary preservation of models.…”

Section: B Quantitative and Qualitative Evaluationsmentioning

confidence: 99%

Neural Contourlet Network for Monocular 360° Depth Estimation

Shen

Lin

Nie

et al. 2022

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

For a monocular 360°image, depth estimation is a challenging because the distortion increases along the latitude. To perceive the distortion, existing methods devote to designing a deep and complex network architecture. In this paper, we provide a new perspective that constructs an interpretable and sparse representation for a 360°image. Considering the importance of the geometric structure in depth estimation, we utilize the contourlet transform to capture an explicit geometric cue in the spectral domain and integrate it with an implicit cue in the spatial domain. Specifically, we propose a neural contourlet network consisting of a convolutional neural network and a contourlet transform branch. In the encoder stage, we design a spatial-spectral fusion module to effectively fuse two types of cues. Contrary to the encoder, we employ the inverse contourlet transform with learned low-pass subbands and band-pass directional subbands to compose the depth in the decoder. Experiments on the three popular 360°panoramic image datasets demonstrate that the proposed approach outperforms the state-of-the-art schemes with faster convergence. Code is available at https://github.com/ zhijieshen-bjtu/Neural-Contourlet-Network-for-MODE.

show abstract

“…Using domain adaptation for synthetic data requires both large-scale 3D models with great variations and corresponding similar 360-degree color ground truth. Most concurrent works [62] [56] either use synthetic datasets (i.e., PanoSunCG [55]) or 3D scanned datasets (i.e., Matterport3D [7], Stanford 2D-3D [2], Pano3D [1]). The former is generated with 3D models and a virtual omnidirectional camera without domain adaptation, and the latter ones are captured with specialized equipment and post-processed.…”

Section: Data In Single-view Depth Estimationmentioning

confidence: 99%

360 Depth Estimation in the Wild -- The Depth360 Dataset and the SegFuse Network

Qi¹,

Shum²,

Morishima³

2022

Preprint

View full text Add to dashboard Cite

Single-view depth estimation from omnidirectional images has gained popularity with its wide range of applications such as autonomous driving and scene reconstruction. Although data-driven learning-based methods demonstrate significant potential in this field, scarce training data and ineffective 360 estimation algorithms are still two key limitations hindering accurate estimation across diverse domains. In this work, we first establish a large-scale dataset with varied settings called Depth360 to tackle the training data problem. This is achieved by exploring the use of a plenteous source of data, 360 videos from the internet, using a test-time training method that leverages unique information in each omnidirectional sequence. With novel geometric and temporal constraints, our method generates consistent and convincing depth samples to facilitate single-view estimation. We then propose an end-to-end two-branch multi-task learning network, SegFuse, that mimics the human eye to effectively learn from the dataset and estimate high-quality depth maps from diverse monocular RGB images. With a peripheral branch that uses equirectangular projection for depth estimation and a foveal branch that uses cubemap projection for semantic segmentation, our method predicts consistent global depth while maintaining sharp details at local regions. Experimental results show favorable performance against the state-of-the-art methods.

show abstract

Pano3D: A Holistic Benchmark and a Solid Baseline for 360° Depth Estimation

Cited by 30 publications

References 37 publications

Hybrid Skip: A Biologically Inspired Skip Connection for the UNet Architecture

Hybrid Skip: A Biologically Inspired Skip Connection for the UNet Architecture

Neural Contourlet Network for Monocular 360° Depth Estimation

360 Depth Estimation in the Wild -- The Depth360 Dataset and the SegFuse Network

Contact Info

Product

Resources

About