Foreground-aware Dense Depth Estimation for 360 Images

Feng, Qi; Shum, Hubert P. H.; Shimamura, Ryo; Morishima, Shigeo

doi:10.24132/jwscg.2020.28.10

Cited by 4 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although this problem is alleviated by a kernel transformer [51] that uses parameterized functions to preserve cross-channel interactions, the model size is still limited. While using equirectangular projection can generate more consistent global prediction due to its wider FOV, small regions with a steep local gradient when regressing the global gradient are harder to learn [14]. Wang et al [56] and Jiang et al [26] use a fusion scheme that combines the depth maps estimated with equirectangular and cubemap projections for sharper depth estimation.…”

Section: Single-view Depth Estimation For Omnidirectional Imagesmentioning

confidence: 99%

360 Depth Estimation in the Wild -- The Depth360 Dataset and the SegFuse Network

Qi¹,

Shum²,

Morishima³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Single-view depth estimation from omnidirectional images has gained popularity with its wide range of applications such as autonomous driving and scene reconstruction. Although data-driven learning-based methods demonstrate significant potential in this field, scarce training data and ineffective 360 estimation algorithms are still two key limitations hindering accurate estimation across diverse domains. In this work, we first establish a large-scale dataset with varied settings called Depth360 to tackle the training data problem. This is achieved by exploring the use of a plenteous source of data, 360 videos from the internet, using a test-time training method that leverages unique information in each omnidirectional sequence. With novel geometric and temporal constraints, our method generates consistent and convincing depth samples to facilitate single-view estimation. We then propose an end-to-end two-branch multi-task learning network, SegFuse, that mimics the human eye to effectively learn from the dataset and estimate high-quality depth maps from diverse monocular RGB images. With a peripheral branch that uses equirectangular projection for depth estimation and a foveal branch that uses cubemap projection for semantic segmentation, our method predicts consistent global depth while maintaining sharp details at local regions. Experimental results show favorable performance against the state-of-the-art methods.

show abstract

Section: Single-view Depth Estimation For Omnidirectional Imagesmentioning

confidence: 99%

360 Depth Estimation in the Wild -- The Depth360 Dataset and the SegFuse Network

Qi¹,

Shum²,

Morishima³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…A great deal of recent work has attempted to understand and process 360°images and videos for better immersive experiences in VR applications. To provide better 3D information for mixed reality applications based on 360°videos, Feng et al [15], [16] and Wang et al [46] proposed deep depth estimation networks working on the spherical domain and built large panorama datasets for training their models. Deep learning techniques have also been used effectively for the semantic understanding of 360°images, including saliency detection [27], object recognition [38], and indoor holistic scene understanding [39].…”

Section: °Image Analysis and Processingmentioning

confidence: 99%

360° Stereo Image Composition with Depth Adaption

Huang¹,

Zhang²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

<p>360° images and videos have become an economic and popular way to provide VR experiences using real-world content. However, the manipulation of the stereo panoramic content remains less explored. In this paper, we focus on the 360° image composition problem, and develop a solution that can take an object from a stereo image pair and insert it at a given 3D position in a target stereo panorama, with well-preserved geometry information. Our method uses recovered 3D point clouds to guide the composited image generation. More specifically, we observe that using only a one-off operation to insert objects into equirectangular images will never produce satisfactory depth perception and generate ghost artifacts when users are watching the result from different view directions. Therefore, we propose a novel view-dependent projection method that segments the object in 3D spherical space with the stereo camera pair facing in that direction. A deep depth densification network is proposed to generate depth guidance for the stereo image generation of each view segment according to the desired position and pose of the inserted object. We finally merge the synthesized view segments and blend the objects into the target stereo 360° scene. A user study demonstrates that our method can provide good depth perception and removes ghost artifacts. The view-dependent solution is a potential paradigm for other content manipulation methods for 360° images and videos. </p>

show abstract