Deep Video Inpainting

Kim, Dahun; Woo, Sanghyun; Lee, Joon-Young; Kweon, In So

doi:10.1109/cvpr.2019.00594

Cited by 177 publications

(127 citation statements)

References 37 publications

Supporting

Mentioning

102

Contrasting

Unclassified

Order By: Relevance

“…Most inpainting algorithms operate offline over the video frames, so that they can refer to and make use of as many frames as possible to find the background and fill in the object mask [51]. VINet [52] is considered one of the state-of-the-art deep learning-based inpainting methods in terms of its computational requirement and accuracy. However, it was not designed for or optimized for small handheld devices.…”

Section: Dynamic Object Removalmentioning

confidence: 99%

“…As for inpainting of the CIRO mask, VINet [52], one of the state-of-the-art of its kind that executes at near real-time, was adapted. VINet is designed as a recurrent network and internally computes the flow fields from five adjacent frames for the target frame.…”

Section: Ciro Diminishing System System Designmentioning

confidence: 99%

“…We compared and evaluated the inpainting performance of the original VINet [52] and our lightweight, real-time version using the structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) [55,56]. SSIM measures the similarity in structure between the two images, and PSNR, the image's distortion.…”

Section: Inpainting Performance Evaluationmentioning

confidence: 99%

See 2 more Smart Citations

CIRO: The Effects of Visually Diminished Real Objects on Human Perception in Handheld Augmented Reality

Kim

Lee

et al. 2021

Electronics

View full text Add to dashboard Cite

Augmented reality (AR) scenes often inadvertently contain real world objects that are not relevant to the main AR content, such as arbitrary passersby on the street. We refer to these real-world objects as content-irrelevant real objects (CIROs). CIROs may distract users from focusing on the AR content and bring about perceptual issues (e.g., depth distortion or physicality conflict). In a prior work, we carried out a comparative experiment investigating the effects on user perception of the AR content by the degree of the visual diminishment of such a CIRO. Our findings revealed that the diminished representation had positive impacts on human perception, such as reducing the distraction and increasing the presence of the AR objects in the real environment. However, in that work, the ground truth test was staged with perfect and artifact-free diminishment. In this work, we applied an actual real-time object diminishment algorithm on the handheld AR platform, which cannot be completely artifact-free in practice, and evaluated its performance both objectively and subjectively. We found that the imperfect diminishment and visual artifacts can negatively affect the subjective user experience.

show abstract

Section: Dynamic Object Removalmentioning

confidence: 99%

Section: Ciro Diminishing System System Designmentioning

confidence: 99%

See 1 more Smart Citation

CIRO: The Effects of Visually Diminished Real Objects on Human Perception in Handheld Augmented Reality

Kim

Lee

et al. 2021

Electronics

View full text Add to dashboard Cite

show abstract

“…Ding et al (Ding et al 2019) and Chang et al (Chang, Liu, and Hsu 2019) focused on exploring the spatial and temporal information with convLSTM layers. Kim et al (Kim et al 2019) enforced the outputs to be temporally consistent by a recurrent feedback. (Xu et al 2019) inpainted all the incomplete optical flows of the video and propagated valid regions to hole regions iteratively.…”

Section: Video Inpaintingmentioning

confidence: 99%

AutoRemover: Automatic Object Removal for Autonomous Driving Videos

Zhang

Peng

et al. 2020

AAAI

View full text Add to dashboard Cite

Motivated by the need for photo-realistic simulation in autonomous driving, in this paper we present a video inpainting algorithm AutoRemover, designed specifically for generating street-view videos without any moving objects. In our setup we have two challenges: the first is the shadow, shadows are usually unlabeled but tightly coupled with the moving objects. The second is the large ego-motion in the videos. To deal with shadows, we build up an autonomous driving shadow dataset and design a deep neural network to detect shadows automatically. To deal with large ego-motion, we take advantage of the multi-source data, in particular the 3D data, in autonomous driving. More specifically, the geometric relationship between frames is incorporated into an inpainting deep neural network to produce high-quality structurally consistent video output. Experiments show that our method outperforms other state-of-the-art (SOTA) object removal algorithms, reducing the RMSE by over 19%.

show abstract

“…CombCN inpaints each frame with a 2D convolution operation and supports the 3DCN to capture the temporal structure. Kim et al [17] have proposed a 3D-2D encoder-decoder network architecture using ConvLSTM for the frame inpainting in a damaged video. They have designed the network to train aggregated feature maps from the past and future frames for inpainting.…”

Section: Related Workmentioning

confidence: 99%

Partial Convolutional LSTM for Spatiotemporal Prediction of Incomplete Data

Son

Jang

2020

IEEE Access

View full text Add to dashboard Cite

Advanced data analysis techniques facilitate data-driven spatiotemporal prediction in various fields. However, in real-world data, missing values are inevitable, which causes the data incomplete and makes predictions more challenging. Although we can train complex spatiotemporal correlations with deep learning techniques, most deep learning networks require data without any missing values. In this paper, we propose a novel deep learning framework that manages missing values in the grid-based data structure. We design a partial convolutional long-short-term-memory (PConvLSTM) by combining partial convolution for inpainting and convolutional long-short-term-memory (ConvLSTM) for spatiotemporal prediction. We treat incomplete spatiotemporal data with the partial convolution and train spatiotemporal dependencies with the ConvLSTM structure. The trained PConvLSTM can predict continuous spatial data with missing regions in incomplete input data. We also train the network using incomplete spatiotemporal data without ground truth to enhance practicality. Existing deep learning networks to interpolate missing data are mostly trained by applying ground truth data without missing regions. We show that PConvLSTM achieves higher prediction accuracies compared to ConvLSTM for incomplete data without ground truth.

show abstract

Deep Video Inpainting

Cited by 177 publications

References 37 publications

CIRO: The Effects of Visually Diminished Real Objects on Human Perception in Handheld Augmented Reality

CIRO: The Effects of Visually Diminished Real Objects on Human Perception in Handheld Augmented Reality

AutoRemover: Automatic Object Removal for Autonomous Driving Videos

Partial Convolutional LSTM for Spatiotemporal Prediction of Incomplete Data

Contact Info

Product

Resources

About