Computational video editing for dialogue-driven scenes

Abstract

We present a system for efficiently editing video of dialogue-driven scenes. The input to our system is a standard film script and multiple video takes, each capturing a different camera framing or performance of the complete scene. Our system then automatically selects the most appropriate clip from one of the input takes, for each line of dialogue, based on a user-specified set of film-editing idioms. Our system starts by segmenting the input script into lines of dialogue and then splitting each input take into a sequence of clips time-aligned with each line. Next, it labels the script and the clips with high-level structural information (e.g., emotional sentiment of dialogue, camera framing of clip, etc.). After this pre-process, our interface offers a set of basic idioms that users can combine in a variety of ways to build custom editing styles. Our system encodes each basic idiom as a Hidden Markov Model that relates editing decisions to the labels extracted in the pre-process. For short scenes (< 2 minutes, 8--16 takes, 6--27 lines of dialogue) applying the user-specified combination of idioms to the pre-processed inputs generates an edited sequence in 2--3 seconds. We show that this is significantly faster than the hours of user time skilled editors typically require to produce such edits and that the quick feedback lets users iteratively explore the space of edit designs.

Autonomous aerial cinematography in unstructured environments with learned artistic decision‐making

“…For example, in Drucker and Zeltzer (), the researchers use an A* planner to move a virtual camera in precomputed indoor simulation scenarios to avoid collisions with obstacles in 2D. More recently, we find works such as Leake, Davis, Truong, and Agrawala () that postprocesses videos of a scene taken from different angles by automatically labeling features of different views. The approach uses high‐level user‐specified rules which exploit the labels to automatically select the optimal sequence of viewpoints for the final movie.…”

Can a Robot Become a Movie Director? Learning Artistic Principles for Aerial Cinematography

“…For example, in [18], the researchers use an A* planner to move a virtual camera in pre-computed indoor simulation scenarios to avoid collisions with obstacles in 2D. More recently, we find works such as [23] that post-processes videos of a scene taken from different angles by automatically labeling features of different views. The approach uses high-level user-specified rules which exploits the formerly labeled features in order to automatically select the optimal sequence of viewpoints for the final movie.…”

Watch to Edit: Video Retargeting using Gaze

“…Leake et al . [LDTA17] propose a computational video editing approach for dialogue‐driven scenes which utilizes the script and multiple video recordings of the scene to select the optimal recording that best satisfies user preferences (such as emphasize a particular character, intensify emotional dialogues, etc .). A more general effort was made by Galvane et al .…”

“…While human attention is influenced by bottom-up cues, it is also impacted by top-down cues relating to scene semantics such as faces, spoken dialogue, scene actions and emotions which are integral to the storyline [SSSM14, GSY * 15]. Leake et al [LDTA17] propose a computational video editing approach for dialoguedriven scenes which utilizes the script and multiple video recordings of the scene to select the optimal recording that best satisfies user preferences (such as emphasize a particular character, intensify emotional dialogues, etc.). A more general effort was made by Galvane et al [GRLC15] for continuity editing in the 3D animated sequences.…”