Joint Stereo Video Deblurring, Scene Flow Estimation and Moving Object Segmentation

Abstract

Stereo videos for the dynamic scenes often show unpleasant blurred effects due to the camera motion and the multiple moving objects with large depth variations. Given consecutive blurred stereo video frames, we aim to recover the latent clean images, estimate the 3D scene flow and segment the multiple moving objects. These three tasks have been previously addressed separately, which fail to exploit the internal connections among these tasks and cannot achieve optimality. In this paper, we propose to jointly solve these three tasks in a unified framework by exploiting their intrinsic connections. To this end, we represent the dynamic scenes with the piece-wise planar model, which exploits the local structure of the scene and expresses various dynamic scenes. Under our model, these three tasks are naturally connected and expressed as the parameter estimation of 3D scene structure and camera motion (structure and motion for the dynamic scenes). By exploiting the blur model constraint, the moving objects and the 3D scene structure, we reach an energy minimization formulation for joint deblurring, scene flow and segmentation. We evaluate our approach extensively on both synthetic datasets and publicly available real datasets with fastmoving objects, camera motion, uncontrolled lighting conditions and shadows. Experimental results demonstrate that our method can achieve significant improvement in stereo video deblurring, scene flow estimation and moving object segmentation, over stateof-the-art methods.

Learning Event-Based Motion Deblurring

“…DAVANet [32] is proposed to handle parallax by bidirectional disparities and varying information from two views to obtain stereo-sharp image pairs. Pan et al, [31] propose a joint optimization framework to restore the latent images for generic dynamic scenes that benefited from incorporating 3D scene cues with pre-estimated scene flow and the improved boundaries information. However, the lack of temporal information prevents the above methods from generating sequences of sharp images.…”

Single Image Optical Flow Estimation with an Event Camera

“…As the flow accuracy highly depends on the quality of the image, a better-restored image also re-lies on the quality of the estimated flow. Researchers attempt to use flow to estimate the spatial-varying blur kernel and then restore images [53,21,22,46,35,36,34]. Recently, learning-based methods have brought significant improvements in image deblurring [19,33,56].…”

U2-ONet: A Two-level Nested Octave U-structure with Multiscale Attention Mechanism for Moving Instances Segmentation

“…This kind of method realizes the result of feature points level instead of pixel-by-pixel, and the application conditions and scenarios are limited because of only segmenting more salient moving object, limited model numbers of segmentation and slow computational speed. More recent approaches propose optical flow based method for motion segmentation, as in [8], [9], [10], [11], [31], [32], [33]. CC [31] introduces a generic framework called Competitive Collaboration (CC), in which four networks including motion segmentation network learn to collaborate and compete, thereby achieving specific goals such as segmenting the scene into moving objects and the static background.…”

“…CC [31] introduces a generic framework called Competitive Collaboration (CC), in which four networks including motion segmentation network learn to collaborate and compete, thereby achieving specific goals such as segmenting the scene into moving objects and the static background. The method [32] takes advantage of the interconnectedness of three tasks including moving object segmentation and solves them jointly in a unified framework. However, both methods cannot segment each object instance.…”