How Not to Be Seen — Object Removal from Videos of Crowded Scenes

Abstract

Figure 1: To remove the foremost person from this video, both the dynamic scene elements and the background behind it need to be restored. In this sample from our Museum sequence, the right-hand-side of each frame pair shows the inpainted result. AbstractRemoving dynamic objects from videos is an extremely challenging problem that even visual effects professionals often solve with time-consuming manual frame-by-frame editing. We propose a new approach to video completion that can deal with complex scenes containing dynamic background and non-periodical moving objects. We build upon the idea that the spatio-temporal hole left by a removed object can be filled with data available on other regions of the video where the occluded objects were visible. Video completion is performed by solving a large combinatorial problem that searches for an optimal pattern of pixel offsets from occluded to unoccluded regions. Our contribution includes an energy functional that generalizes well over different scenes with stable parameters, and that has the desirable convergence properties for a graph-cut-based optimization. We provide an interface to guide the completion process that both reduces computation time and allows for efficient correction of small errors in the result. We demonstrate that our approach can effectively complete complex, high-resolution occlusions that are greater in difficulty than what existing methods have shown.

Visual privacy protection methods: A survey

“…Therefore, these methods can be used for visual privacy to remove people that do not commit suspicious activities from video surveillance recordings, removing inactive participants from the video stream of a video conference, concealing people from images in online social networks, and so many other applications. However, due to computational restrictions, inpainting methods are rarely used in real-time applications running on commodity hardware (Granados et al, 2012).…”

Dense motion estimation between distant frames: Combinatorial multi-step integration and statistical selection

“…Our experiments focus on frame pairs taken from three sequences: MPI S1 [14], Hope and Newspaper. For the selected pairs, the combinatorial multi-step integration has been performed taking input elementary flow fields estimated with a 2D version of the disparity estimator of [17].…”

“…Fig. 3: Source frames of the MPI S1 sequence [14] and reconstruction of the kiosk of I 46 from I 25 with: 1) the statistical processing (SP), 2) the global optimization (GO) method solved by fusion moves [15], 3) both combined (SP+GO).…”

Dense Long-term Motion Estimation via Statistical Multi-step Flow

“…Our experiments focus on the following sequences: MPI S1 (Granados et al, 2012) Fig.4 and 6a-h, Hope Fig.6i-p, Newspaper Fig.6q-t, Walking Couple Fig.7 and Flag (Garg et al, 2013) Fig.8. The proposed statistical multi-step flow is referred to as StatFlow in the following.…”

“…• direct estimation between each pair {I re f , I n } using LDOF (Brox and Malik, 2011), ITV-L1 ) and the keypoint-based nonrigid registration of (Pizarro and Bartoli, 2012), • concatenation of optical flows computed between consecutive frames using LDOF (LDOF acc), : Source frames of the MPI S1 sequence (Granados et al, 2012) and reconstruction of the kiosk of I 25 from I 45 with: e) the combinatorial integration and the statistical selection introduced in (Conze et al, 2013), f) the proposed extended version described in Section 2.1 (initial phase of StatFlow). Black boxes focus on differences between both methods.…”