Short-Term and Long-Term Context Aggregation Network for Video Inpainting

Abstract

Video inpainting aims to restore missing regions of a video and has many applications such as video editing and object removal. However, existing methods either suffer from inaccurate short-term context aggregation or rarely explore long-term frame information. In this work, we present a novel context aggregation network to effectively exploit both short-term and long-term frame information for video inpainting. In the encoding stage, we propose boundary-aware shortterm context aggregation, which aligns and aggregates, from neighbor frames, local regions that are closely related to the boundary context of missing regions into the target frame 5 . Furthermore, we propose dynamic long-term context aggregation to globally refine the feature map generated in the encoding stage using long-term frame features, which are dynamically updated throughout the inpainting process. Experiments show that it outperforms state-of-the-art methods with better inpainting results and fast inpainting speed.

Flow-Guided Video Inpainting with Scene Templates

“…To hallucinate flow inside the masked region, non-learning approaches [22,7] rely on energy minimization assuming smoothness of the flow; [29,5] is a deep learning solution that first computes flow between image pairs, then uses a neural network to hallucinate flow inside the masked region. End-to-end learning methods [11,26,19,38,2,37,15] model cross-frame correspondence in their loss functions. For example, [38] jointly infers appearance and flow while penalizing temporal inconsistency.…”

Optical Flow Correction Network for Repairing Video

“…These techniques concentrate on repairing single-frame images and wonderful results have been achieved. Some recent research results [10][11][12], in particular, have shown that deep learning methods have excellent repair capabilities. Furthermore, video repair and related technologies can also be used for tasks such as eliminating watermarks [13], object removal [14], and movie special effects [15].…”

Optical flow correction network for repairing video

“…These techniques concentrate on repairing single-frame images and wonderful results have been achieved. Some recent research results [10][11][12], in particular, have shown that deep learning methods have excellent repair capabilities. Furthermore, video repair and related technologies can also be used for tasks such as eliminating watermarks [13], object removal [14], and movie special effects [15].…”