Boosted GAN with Semantically Interpretable Information for Image Inpainting

Li, Ang; Qi, Jianzhong; Zhang, Rui; Ramamohanarao, Kotagiri

doi:10.1109/ijcnn.2019.8851926

Cited by 12 publications

(6 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The generative adversarial network (GAN) was introduced by Goodfellow et al [5] to produce realistic images under certain conditions. GANs have attracted substantial attention and has been studied in many tasks [17], such as image synthesis [5,13,14,21], text-to-image translation [22,36], and image inpainting [11,15,16]. In this work, we focus on talking head video generation with GAN guided by 3D facial depth maps learned without any ground-truth depths.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Depth-Aware Generative Adversarial Network for Talking Head Video Generation

Hong¹,

Zhang²,

Shen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Talking head video generation aims to produce a synthetic human face video that contains the identity and pose information respectively from a given source image and a driving video. Existing works for this task heavily rely on 2D representations (e.g. appearance and motion) learned from the input images. However, dense 3D facial geometry (e.g. pixel-wise depth) is extremely important for this task as it is particularly beneficial for us to essentially generate accurate 3D face structures and distinguish noisy information from the possibly cluttered background. Nevertheless, dense 3D geometry annotations are prohibitively costly for videos and are typically not available for this video generation task. In this paper, we first introduce a self-supervised geometry learning method to automatically recover the dense 3D geometry (i.e. depth) from the face videos without the requirement of any expensive 3D annotation data. Based on the learned dense depth maps, we further propose to leverage them to estimate sparse facial keypoints that capture the critical movement of the human head. In a more dense way, the depth is also utilized to learn 3D-aware cross-modal (i.e. appearance and depth) attention to guide the generation of motion fields for warping source image representations. All these contributions compose a novel depth-aware generative adversarial network (DaGAN) for talking head generation. Extensive experiments conducted demonstrate that our proposed method can generate highly realistic faces, and achieve significant results on the unseen human faces. 1

show abstract

Section: Related Workmentioning

confidence: 99%

“…For the optimization losses, we set λ P = 10, λ G =1, λ E = 10, and λ D = 10. We set the number of keypoints in DaGAN as 15 stage, we first train our face depth network using consecutive frames from videos in VoxCeleb1, and we fix it during the training of the whole deep generation framework.…”

Section: Implementation Detailsmentioning

confidence: 99%

Depth-Aware Generative Adversarial Network for Talking Head Video Generation

Hong¹,

Zhang²,

Shen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Different from conventional image inpainting approaches, deep learning-based image inpainting models can generate more visually plausible details or fill large missing regions with new contents that never exist in the input image [Pathak et al, 2016;Iizuka et al, 2017;Yu et al, 2018;Liu et al, 2018;Li et al, 2019b;Liu et al, 2019;Li et al, 2019a;Li et al, 2020;Zeng et al, 2021]…”

Section: Image Inpaintingmentioning

confidence: 99%

Iterative Geometry-Aware Cross Guidance Network for Stereo Image Inpainting

Li¹,

Song²,

Zhang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Currently, single image inpainting has achieved promising results based on deep convolutional neural networks. However, inpainting on stereo images with missing regions has not been explored thoroughly, which is also a significant but different problem. One crucial requirement for stereo image inpainting is stereo consistency. To achieve it, we propose an Iterative Geometry-Aware Cross Guidance Network (IGGNet). The IGGNet contains two key ingredients, i.e., a Geometry-Aware Attention (GAA) module and an Iterative Cross Guidance (ICG) strategy. The GAA module relies on the epipolar geometry cues and learns the geometryaware guidance from one view to another, which is beneficial to make the corresponding regions in two views consistent. However, learning guidance from co-existing missing regions is challenging. To address this issue, the ICG strategy is proposed, which can alternately narrow down the missing regions of the two views in an iterative manner. Experimental results demonstrate that our proposed network outperforms the latest stereo image inpainting model and state-of-the-art single image inpainting models.

show abstract

“…These methods often suffer from low generation quality, especially when dealing with complicated scenes or large missing regions [8,19]. Image inpaint methods [8,14,15,16,19,21,27,30,31,32,33] with deep learning techniques have attracted wide attention. Pathak et al [19] introduce the Context Encoder (CE) model where a convolutional encoder-decoder network is trained with the combination of an adversarial loss [6] and a reconstruction loss.…”

Section: Image Inpaintingmentioning

confidence: 99%

Short-Term and Long-Term Context Aggregation Network for Video Inpainting

Zhao

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Video inpainting aims to restore missing regions of a video and has many applications such as video editing and object removal. However, existing methods either suffer from inaccurate short-term context aggregation or rarely explore long-term frame information. In this work, we present a novel context aggregation network to effectively exploit both short-term and long-term frame information for video inpainting. In the encoding stage, we propose boundary-aware shortterm context aggregation, which aligns and aggregates, from neighbor frames, local regions that are closely related to the boundary context of missing regions into the target frame 5 . Furthermore, we propose dynamic long-term context aggregation to globally refine the feature map generated in the encoding stage using long-term frame features, which are dynamically updated throughout the inpainting process. Experiments show that it outperforms state-of-the-art methods with better inpainting results and fast inpainting speed.

show abstract

Boosted GAN with Semantically Interpretable Information for Image Inpainting

Cited by 12 publications

References 37 publications

Depth-Aware Generative Adversarial Network for Talking Head Video Generation

Depth-Aware Generative Adversarial Network for Talking Head Video Generation

Iterative Geometry-Aware Cross Guidance Network for Stereo Image Inpainting

Short-Term and Long-Term Context Aggregation Network for Video Inpainting

Contact Info

Product

Resources

About