Directing Cinematographic Drones

Galvane, Quentin; Lino, Christophe; Christie, Marc; Fleureau, Julien; Servant, Fabien; Tariolle, François-Louis; Guillotel, Philippe

doi:10.1145/3181975

Cited by 57 publications

(59 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This environment offers a decent opportunity to evaluate the performance of the trained agent as it has a clear pattern of blocks that the drone can avoid. Our agent performs significantly better than a random policy in the training environment, as well as in the two previously unknown testing environments (Table I) 3 . Our agent is able to achieve all goals that it was supposed to learn:…”

Section: Resultsmentioning

confidence: 82%

Can a Robot Become a Movie Director? Learning Artistic Principles for Aerial Cinematography

Gschwindt

Camci

Bonatti

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Aerial filming is constantly gaining importance due to the recent advances in drone technology. It invites many intriguing, unsolved problems at the intersection of aesthetical and scientific challenges. In this work, we propose a deep reinforcement learning agent which supervises motion planning of a filming drone by making desirable shot mode selections based on aesthetical values of video shots. Unlike most of the current state-of-the-art approaches that require explicit guidance by a human expert, our drone learns how to make favorable viewpoint selections by experience. We propose a learning scheme that exploits aesthetical features of retrospective shots in order to extract a desirable policy for better prospective shots. We train our agent in realistic AirSim simulations using both a hand-crafted reward function as well as reward from direct human input. We then deploy the same agent on a real DJI M210 drone in order to test the generalization capability of our approach to real world conditions. To evaluate the success of our approach in the end, we conduct a comprehensive user study in which participants rate the shot quality of our methods. Videos of the system in action can be seen at https://youtu.be/qmVw6mfyEmw.

show abstract

Section: Resultsmentioning

confidence: 82%

Can a Robot Become a Movie Director? Learning Artistic Principles for Aerial Cinematography

Gschwindt

Camci

Bonatti

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

show abstract

“…In particular, we analyze the overhead of the OLSC Solver within the system and establish the feasibility of adopting this enhanced model in a real surveillance system. e prototype we employ for this evaluation leverages typical hardware that can be found at most research labs and is comparable in scale to the experiments reported on closely related systems (Galvane et al 2017;Nägeli et al 2017). In addition, we present a set of large scale simulations that compare the performance of the proposed algorithms against a baseline and establish the sampling heuristic as the method of choice, which we employ on the prototype.…”

Section: Discussionmentioning

confidence: 99%

“…Autonomous cinematography is another application for drones, beyond coverage and tracking, the aesthetic quality plays a key role in viewpoint planning (Joubert et al 2016). Building upon earlier work in virtual cinematography (Lino et al 2011), this exciting line of work has recently been witnessing very interesting developments (Galvane et al 2017;Joubert et al 2015;Nägeli et al 2017). In earlier work, we developed several target coverage algorithms for targets represented as points and deployed them on our testbed (Khan et al 2016;Neishaboori et al 2014a,b;Saeed et al 2014).…”

Section: Related Workmentioning

confidence: 99%

“…To the best of our knowledge, no earlier work in smart surveillance tackled these challenges simultaneously. Very recently, (Galvane et al 2017;Nägeli et al 2017) presented novel systems for automated cinematography using drones, taking into account framing objectives, occlusions and collision-avoidance. However, they focus on scenarios involving a known number of actors where a user speci es the desired location of each actor in the camera frame.…”

mentioning

confidence: 99%

See 1 more Smart Citation

On Realistic Target Coverage by Autonomous Drones

Ahmed

Abdelkader

Khan

et al. 2019

ACM Trans. Sen. Netw.

View full text Add to dashboard Cite

Low-cost mini-drones with advanced sensing and maneuverability enable a new class of intelligent sensing systems. To achieve the full potential of such drones, it is necessary to develop new enhanced formulations of both common and emerging sensing scenarios.Namely, several fundamental challenges in visual sensing remain unsolved including: 1) Fi ing sizable targets in camera frames; 2) E ective viewpoints matching target poses; 3) Occlusion by elements in the environment, including other targets. In this paper, we introduce Argus: an autonomous system that utilizes drones to incrementally collect target information through a two-tier architecture.To tackle the stated challenges, Argus employs a novel geometric model that captures both target shapes and coverage constraints.Recognizing drones as the scarcest resource, Argus aims to minimize the number of drones required to cover a set of targets. We prove this problem is NP-hard, and even hard to approximate, before deriving a best-possible approximation algorithm along with a competitive sampling heuristic which runs up to 100x faster according to large-scale simulations. To test Argus in action, we demonstrate and analyze its performance on a prototype implementation. Finally, we present a number of extensions to accommodate more application requirements and highlight some open problems.

show abstract

“…When dealing with dynamic targets, the UAV must anticipate the actor's motion and reason about collisions and occlusions generated by obstacles in potential trajectories. The challenge of visibility has been explored in previous works in varying degrees of obstacle complexity (Bonatti, Ho, Wang, Choudhury, & Scherer, ; Galvane et al, ; Nägeli et al, ; Penin, Giordano, & Chaumette, ).…”

Section: Introductionmentioning

confidence: 99%

Autonomous aerial cinematography in unstructured environments with learned artistic decision‐making

Bonatti

Wang

et al. 2020

Journal of Field Robotics

View full text Add to dashboard Cite

Aerial cinematography is revolutionizing industries that require live and dynamic camera viewpoints such as entertainment, sports, and security. However, safely piloting a drone while filming a moving target in the presence of obstacles is immensely taxing, often requiring multiple expert human operators. Hence, there is a demand for an autonomous cinematographer that can reason about both geometry and scene context in real-time. Existing approaches do not address all aspects of this problem; they either require high-precision motion-capture systems or global positioning system tags to localize targets, rely on prior maps of the environment, plan for short time horizons, or only follow fixed artistic guidelines specified before the flight. In this study, we address the problem in its entirety and propose a complete system for real-time aerial cinematography that for the first time combines: (a) vision-based target estimation; (b) 3D signed-distance mapping for occlusion estimation; (c) efficient trajectory optimization for long time-horizon camera motion; and (d) learning-based artistic shot selection.We extensively evaluate our system both in simulation and in field experiments by filming dynamic targets moving through unstructured environments. Our results indicate that our system can operate reliably in the real world without restrictive assumptions. We also provide in-depth analysis and discussions for each module, with the hope that our design tradeoffs can generalize to other related applications. Videos of the complete system can be found at https://youtu.be/ookhHnqmlaU. K E Y W O R D S aerial robotics, cinematography, computer vision, learning, mapping, motion planning Within the filming context, this cost function measures jerkiness of motion, safety, environmental occlusion of the actor, and shot quality (artistic quality of viewpoints). This cost function depends on the environment and , and on the actor forecast ξ a , all of which are sensed on-the-fly. The changing nature of the environment and ξ a demands replanning at a high frequency. Here we briefly touch upon the four components of the cost function J(ξ q ) (refer to Section 7 for details and mathematical expressions): Smoothness J smooth (ξ q ): Penalizes jerky motions that may lead to camera blur and unstable flight; Safety J obs (ξ q , ): Penalizes proximity to obstacles that are unsafe for the UAV; Occlusion J occ (ξ q , ξ a , ): Penalizes occlusion of the actor by obstacles in the environment; Shot quality J shot (ξ q , ξ a , Ω art ): Penalizes poor viewpoint angles and scales that deviate from the desired artistic guidelines, given by the set of parameters Ω art . In its simplest form, we can express J(ξ q ) as a linear composition of each individual cost, weighted by scalars λ i . The objective is to * = ( ) ()={ } J J J J J J xyz J J J J J J x y z J occ + J obs 99.4 ± 2.2 94.2 ± 7.3 86.9 ± 9.3 J obs 98.8 ± 3.0 87.1 ± 8.5 75.3 ± 11.8 Avg. dist. to ξ shot (m) J occ + J obs 0.4 ± 0.4 6.2 ± 11.2 10.7 ± 13.2 J obs 0.05 ± 0.1 0.3 ± 0.2 0.5 ± 0...

show abstract

Directing Cinematographic Drones

Cited by 57 publications

References 23 publications

Can a Robot Become a Movie Director? Learning Artistic Principles for Aerial Cinematography

Can a Robot Become a Movie Director? Learning Artistic Principles for Aerial Cinematography

On Realistic Target Coverage by Autonomous Drones

Autonomous aerial cinematography in unstructured environments with learned artistic decision‐making

Contact Info

Product

Resources

About