Example-driven virtual cinematography by learning camera behaviors

Abstract

Designing a camera motion controller that has the capacity to move a virtual camera automatically in relation with contents of a 3D animation, in a cinematographic and principled way, is a complex and challenging task. Many cinematographic rules exist, yet practice shows there are significant stylistic variations in how these can be applied. In this paper, we propose an example-driven camera controller which can extract camera behaviors from an example film clip and re-apply the extracted behaviors to a 3D animation, through learning from a collection of camera motions. Our first technical contribution is the design of a low-dimensional cinematic feature space that captures the essence of a film's cinematic characteristics (camera angle and distance, screen composition and character configurations) and which is coupled with a neural network to automatically extract these cinematic characteristics from real film clips. Our second technical contribution is the design of a cascaded deep-learning architecture trained to (i) recognize a variety of camera motion behaviors from the extracted cinematic features, and (ii) predict the future motion of a virtual camera given a character 3D animation. We propose to rely on a Mixture of Experts (MoE) gating+prediction mechanism to ensure that distinct camera behaviors can be learned while ensuring generalization. We demonstrate the features of our approach through experiments that highlight (i) the quality of our cinematic feature extractor (ii) the capacity to learn a range of behaviors through the gating mechanism, and (iii) the ability to generate a variety of camera motions by applying different behaviors extracted from film clips. Such an example-driven approach offers a high level of controllability which opens new possibilities toward a deeper understanding of cinematographic style and enhanced possibilities in exploiting real film data in virtual environments.

Camera keyframing with style and control

“…The movie data is extracted from the MovieNet dataset [19] which consists of 1,100 movies and 1,600,000 clips. We estimated the cinematic features from a subset of movie clips using the cinematic feature estimator [21] (a convolutional neural network which regresses the cinematic features from 2D skeleton motions). For the movie dataset, we first filter the sequences according to their number of characters and clip length.…”

“…Collision avoidance Collision avoidance is always a critical issue when artists design animations since the desired camera motion may unexpectedly hit other objects in the virtual scene. Compared with the example-driven solution [21], which has to redundantly try multiple references clips to find one without collision, here the designer has the ability to avoid collision by easily inserting a keyframe or forcing the velocity to where no obstacles blind the camera's view. Frames with red camera icon at the corner refer to keyframe constraints.…”

“…To address ambiguities (different camera trajectories for similar character motions or actions), approaches have either tagged motions by hand according to styles of motion [16], used reinforcement techniques [4] or one-shot learning [15]. In the general case of virtual cinematography, a possibility only recently explored [21] is to mimic camera behaviors extracted from film clips. A camera behavior is defined as a temporal correlation between camera trajecto-ries and character/actor features such as distance, relative orientation and size.…”

Optimal Control to Support High-Level User Goals in Human-Computer Interaction

“…automated camera control in virtual environments: Automated camera placement (Lino & Christie, 2012) and motion planning (Li & Cheng, 2008;Yeh et al, 2011) have been studied extensively in virtual environments (Christie, Olivier, & Normand, 2008). A very recent work uses a deep learning approach to learn automated camera control from real film sequences (Jiang et al, 2020). These works share our goal of assisting users in the creation of camera motion and introduced the idea to define viewing constraints in image space (c.f., (Drucker & Zeltzer, 1994;Gleicher & Witkin, 1992;Lino & Christie, 2015;Lino et al, 2011)).…”

Multi-robot Task Assignment for Aerial Tracking with Viewpoint Constraints

“…These works explore parameterizing desired shot qualities and controls [2][3][4] [5], feasibility of dynamic shots [6], and assigning sequences of shots [7] [8]. Other work as focused on algorithmic frameworks for helping directors achieve desirable aesthetic qualities of their shots [9] [10].…”