Internal Video Inpainting by Implicit Long-range Propagation

Ouyang, Hao; Wang, Tengfei; Chen, Qifeng

doi:10.48550/arxiv.2108.01912

“…Subsequently, the video inpainting model processes the RGB images and mask images obtained from the CNN network to generate restored images. The image restoration model we utilize is based on IIVI from [30], which produces high-quality restored images in scenarios with minimal background changes. In the practical application of SLAM systems, scenarios often involve significant changes in the scene, its immediate use for localization and map reconstruction is not feasible; hence, evaluation is essential.…”

Section: System Descriptionmentioning

confidence: 99%

IVI-SLAM: Tracking, Mapping by Internal Video Inpainting for Dynamic Scene

Hua,

Liu,

Zhang

et al. 2024

Artificial Intelligence and Human-Computer Interaction

0

View full text Add to dashboard Cite

In visual SLAM algorithms, the assumption of scene rigidity serves as the foundation for algorithm operation. However, this assumption restricts the use of most visual SLAM systems in densely populated or vehicle-intensive environments, limiting their application in scenarios involving service robots or autonomous driving. In this article, we introduce IVI-SLAM, which is a visual SLAM system based on DynaSLAM. IVI-SLAM extends DynaSLAM by adding background restoration capabilities for both monocular and stereo modes, along with a method for depth recovery. We detect dynamic objects by using deep learning techniques and restore the background occluded by these dynamic objects by iterative training of image inpainting models. Subsequently, we utilize a depth recovery approach to restore the depth values in the affected region. We evaluate our system on publicly available monocular and stereo datasets, achieving promising results.

show abstract

“…Some recent research results [10][11][12], in particular, have shown that deep learning methods have excellent repair capabilities. Furthermore, video repair and related technologies can also be used for tasks such as eliminating watermarks [13], object removal [14], and movie special effects [15].…”

Section: Introductionmentioning

confidence: 99%

Optical Flow Correction Network for Repairing Video

Huang

¹

,

Luo

²

2023

Preprint

0

View full text Add to dashboard Cite

Some areas in the video may be defective due to storage medium damage or data transmission loss. The defective area can be repaired utilizing the optical flow since the optical flow represents the motion information of the pixel and contains the information redundancy properties of the video. However, it is a challenge for current algorithms to directly calculate the correct optical flow in the defective area. Therefore, an optical flow correction network (OFC-Net) is proposed to solve this problem. Firstly, as the basic framework, OFC-Net employs a lightweight U-shaped structure to gradually "repair" the broken optical flow map based on the known information in the defective neighborhood. And then the activation function that conforms to the property of optical flow is used to accelerate the model's convergence during training, while the simplified loss function is adopted to evaluate the similarity of the optical flow map from a global perspective to improve the prediction accuracy. As a result, the corrected optical flow output at the masked position can be obtained. Secondly, the optical flow corrected by OFC-Net can guide locating the pixels in nearby frames to repair the defective region. Optical flow correction and video repair experiments were conducted on MPI Sintel, DAVIS datasets, and real movie clips. The experimental results show that the corrected optical flow map is accurate without bringing artifacts under a wide range of scenarios and the arbitrary irregular mask shapes, which demonstrate the network's strong generalization capabilities. The average PSNR and SSIM of repaired video frames are greater than 33.67 and 0.986, respectively, outperforming the results from the traditional method PatchMatch and deep learning-based synthetic pixel methods.

show abstract

“…Some recent research results [10][11][12], in particular, have shown that deep learning methods have excellent repair capabilities. Furthermore, video repair and related technologies can also be used for tasks such as eliminating watermarks [13], object removal [14], and movie special effects [15].…”

Section: Introductionmentioning

confidence: 99%

Optical flow correction network for repairing video

Huang

¹

,

Luo

²

2023

SIViP

0

View full text Add to dashboard Cite

Some areas in the video may be defective due to storage medium damage or data transmission loss. The defective area can be repaired utilizing the optical flow since the optical flow represents the motion information of the pixel and contains the information redundancy properties of the video. However, it is a challenge for current algorithms to directly calculate the correct optical flow in the defective area. Therefore, an optical flow correction network (OFC-Net) is proposed to solve this problem. Firstly, as the basic framework, OFC-Net employs a lightweight U-shaped structure to gradually "repair" the broken optical flow map based on the known information in the defective neighborhood. And then the activation function that conforms to the property of optical flow is used to accelerate the model's convergence during training, while the simplified loss function is adopted to evaluate the similarity of the optical flow map from a global perspective to improve the prediction accuracy. As a result, the corrected optical flow output at the masked position can be obtained. Secondly, the optical flow corrected by OFC-Net can guide locating the pixels in nearby frames to repair the defective region. Optical flow correction and video repair experiments were conducted on MPI Sintel, DAVIS datasets, and real movie clips. The experimental results show that the corrected optical flow map is accurate without bringing artifacts under a wide range of scenarios and the arbitrary irregular mask shapes, which demonstrate the network's strong generalization capabilities. The average PSNR and SSIM of repaired video frames are greater than 33.67 and 0.986, respectively, outperforming the results from the traditional method PatchMatch and deep learning-based synthetic pixel methods.

show abstract

Internal Video Inpainting by Implicit Long-range Propagation

Cited by 3 publications

References 34 publications

IVI-SLAM: Tracking, Mapping by Internal Video Inpainting for Dynamic Scene

IVI-SLAM: Tracking, Mapping by Internal Video Inpainting for Dynamic Scene

Optical Flow Correction Network for Repairing Video

Optical flow correction network for repairing video

Contact Info

Product

Resources

About