Key-Frame-Based Background Sprite Generation for Hole Filling in Depth Image-Based Rendering

Lie, Wen-Nung; Hsieh, Chia-Yung; Lin, Guo-Shiang

doi:10.1109/tmm.2017.2763319

Cited by 19 publications

(10 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…a) Semantic Image Completion: First proposed by Pathak et al [10], Semantic Image Completion (SIC) [10], [13], [26], [17], [27], [12], [18], [15], [16] exploits information from the whole dataset besides the unmasked. There are works [28], [29], [30], [31], [32] exploiting extra reference images, e.g., other frames in the same video, or additional depth information as the semantic context. The Semantic in existing works refers to the image semantics or high-level context in the dataset.…”

Section: Related Workmentioning

confidence: 99%

Grounded and Controllable Image Completion by Incorporating Lexical Semantics

Zhang

Jiang

Huang

et al. 2020

Preprint

View full text Add to dashboard Cite

In this paper, we present an approach, namely Lexical Semantic Image Completion (LSIC), that may have potential applications in art, design, and heritage conservation, among several others. Existing image completion procedure is highly subjective by considering only visual context, which may trigger unpredictable results which are plausible but not faithful to a grounded knowledge. To permit both grounded and controllable completion process, we advocate generating results faithful to both visual and lexical semantic context, i.e., the description of leaving holes or blank regions in the image (e.g., hole description). One major challenge for LSIC comes from modeling and aligning the structure of visual-semantic context and translating across different modalities. We term this process as structure completion, which is realized by multigrained reasoning blocks in our model. Another challenge relates to the unimodal biases, which occurs when the model generates plausible results without using the textual description. This can be true since the annotated captions for an image are often semantically equivalent in existing datasets, and thus there is only one paired text for a masked image in training. We devise an unsupervised unpaired-creation learning path besides the overexplored paired-reconstruction path, as well as a multi-stage training strategy to mitigate the insufficiency of labeled data. We conduct extensive quantitative and qualitative experiments as well as ablation studies, which reveal the efficacy of our proposed LSIC.

show abstract

Section: Related Workmentioning

confidence: 99%

Grounded and Controllable Image Completion by Incorporating Lexical Semantics

Zhang

Jiang

Huang

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…DIBR is an essential technique for converting 2D images to 3D images. (1)(2)(3)(4) Different gradients or gradient vector flows (GVFs) of the depth in images need different filling techniques. (5)(6)(7)(8)(9)(10) Cheng et al proposed an algorithm for a hypothesized depth gradient model and used a bilateral filter to generate visually comfortable depth maps and diminish block artifacts.…”

Section: Introductionmentioning

confidence: 99%

“…Lie et al proposed a DIBR scheme with a background sprite model for disocclusion and hole filling for 3D TV. (3) A hole-filling method for an edge-oriented morphological operation of DIBR was proposed by Yang et al (4) They applied a bilateral filter as a smooth filter in DIBR and found occlusion regions by edge detection in the depth map to maintain the depth structure. However, DIBR of a virtual image creates image displacement such as holes and distortions; thus, DIBR requires significant improvement.…”

Section: Introductionmentioning

confidence: 99%

Hole Filling in Image Conversion Using Weighted Local Gradients

Lee¹,

Chang²,

Chen³

et al. 2021

Sensors and Materials

View full text Add to dashboard Cite

Three-dimensional displays have become an important medium. Most of the 3D content for these displays is converted from 2D content. The conversion requires appropriate depth information from 2D images, but the information is insufficient to create high-quality 3D images in many cases. Therefore, converting 2D images into 3D images has become an important issue in emerging 3D applications. As even sophisticated equipment produces noise, cracks, and holes in the converted images, the conversion requires an improved method for creating high-quality 3D images, especially a hole-filling method. This method is one of the techniques in depth image-based rendering (DIBR), which converts 2D single-depth images into 3D images by using a multiangle virtual view and stereoscopic image generation. We propose a new algorithm for hole filling in DIBR that converts 2D images into 3D images by maintaining the depth structure of occlusions and performing morphological operations on images and the background depth levels. This algorithm uses edge information, morphology, gradient vector flow, and weighted local gradients to render holes. Multiview images generated by the proposed algorithm have synthesized virtual views of better quality than those generated by previous methods.

show abstract

“…machine vision applications, including scene reconstruction [8,23,27,30,31,33,34,36,44], view synthesis [20,26], navigation [42,43,46], and tracking [3,21].…”

Section: Introductionmentioning

confidence: 99%

Dense Depth Estimation from Multiple 360-degree Images Using Virtual Depth

Yang,

Kim,

Lee

2021

Preprint

View full text Add to dashboard Cite

In this paper, we propose a dense depth estimation pipeline for multiview 360°images. The proposed pipeline leverages a spherical camera model that compensates for radial distortion in 360°images. The key contribution of this paper is the extension of a spherical camera model to multiview by introducing a translation scaling scheme. Moreover, we propose an effective dense depth estimation method by setting virtual depth and minimizing photonic reprojection error. We validate the performance of the proposed pipeline using the images of natural scenes as well as the synthesized dataset for quantitive evaluation. The experimental results verify that the proposed pipeline improves estimation accuracy compared to the current state-of-art dense depth estimation methods.

show abstract

Key-Frame-Based Background Sprite Generation for Hole Filling in Depth Image-Based Rendering

Cited by 19 publications

References 23 publications

Grounded and Controllable Image Completion by Incorporating Lexical Semantics

Grounded and Controllable Image Completion by Incorporating Lexical Semantics

Hole Filling in Image Conversion Using Weighted Local Gradients

Dense Depth Estimation from Multiple 360-degree Images Using Virtual Depth

Contact Info

Product

Resources

About