Watch to Edit: Video Retargeting using Gaze

Abstract

We present a novel approach to optimally retarget videos for varied displays with differing aspect ratios by preserving salient scene content discovered via eye tracking. Our algorithm performs editing with cut, pan and zoom operations by optimizing the path of a cropping window within the original video while seeking to (i) preserve salient regions, and (ii) adhere to the principles of cinematography. Our approach is (a) content agnostic as the same methodology is employed to re‐edit a wide‐angle video recording or a close‐up movie sequence captured with a static or moving camera, and (b) independent of video length and can in principle re‐edit an entire movie in one shot. Our algorithm consists of two steps. The first step employs gaze transition cues to detect time stamps where new cuts are to be introduced in the original video via dynamic programming. A subsequent step optimizes the cropping window path (to create pan and zoom effects), while accounting for the original and new cuts. The cropping window path is designed to include maximum gaze information, and is composed of piecewise constant, linear and parabolic segments. It is obtained via L(1) regularized convex optimization which ensures a smooth viewing experience. We test our approach on a wide variety of videos and demonstrate significant improvement over the state‐of‐the‐art, both in terms of computational complexity and qualitative aspects. A study performed with 16 users confirms that our approach results in a superior viewing experience as compared to gaze driven re‐editing [JSSH15] and letterboxing methods, especially for wide‐angle static camera recordings.

AI Video Editing: a Survey

“…Jain et al [97] also crops movies. [98] aslo edits videos to different ratios. Khoenkaw et al [99] use the film gammer of each shot to guide cinematic feature extraction and generate the importance map at server end.…”

A Fast Smart-Cropping Method and Dataset for Video Retargeting

“…The video aspect ratio transformation algorithms of the literature can be divided in three main categories: a) warping [1,2], b) cropping [3,4,5,6], and c) seam carving [7,8].…”

“…Additional effort is taken to ensure the smooth motion of this crop window throughout the video (e.g. in [3] camera operations are derived by optimizing the path of this window, seeking to adhere to the principles of cinematography). In [13] a Structural Similarity feature is proposed based on blur detection to identify whether an image contains a blurred background.…”

“…In [13] a Structural Similarity feature is proposed based on blur detection to identify whether an image contains a blurred background. In [5] low-level features are employed while in [4,3] eye-gaze information is utilized. In Google's AutoFlip [14], a solution to smart video reframing, face and object detection results are employed -note that this is the only method, to our knowledge, for which its source code is publicly available.…”

Joint Attention for Automated Video Editing

“…Gandhi et al [11] proposed a method to automatically generate edited clips of stage performances recorded by single cameras. Audience gaze data has also been popularly used to determine important elements in the shots, and create a re-edit focusing on the important gaze points [14,24]. These methods are targeted towards re-editing a still camera, adding camera movements that embody the focus of a complex performance (e.g.…”