Multi-camera system for depth based visual effects and compositing

Abstract

Post-production technologies like visual effects (VFX) or digital grading are essential for visual storytelling in today's movie production. Tremendous efforts are made to seamlessly integrate computer generated (CG) elements into live action plates, to refine looks and to provide various delivery formats such as Stereo 3D (S3D). Thus, additional tools to assist and improve element integration, grading and S3D techniques could ease complicated, time consuming, manual and costly processes. Geometric data like depth information is a key for these tasks but a typical main unit camera shoot does not deliver this information. Although e.g. light detection and ranging (LIDAR) scans are used to capture 3D set information, frame by frame geometric information for dynamic scenes including moving camera, actors and props are not being recorded. Stereo camera systems deliver additional data for depth map generation but accuracy is limited. This work suggests a method of capturing light field data within a regular live action shoot. We compute geometric data for post-production use cases such as relighting, depth-based compositing, 3D integration, virtual camera, digital focus as well as virtual backlots. Thus, the 2D high quality life action plate shot by the Director of Photography (DoP) comes along with image data from the light field array that can be further processed and utilized. As a proof-of-concept we produced a fictitious commercial using a main camera and a multi-camera array fitted into an S3D mirror rig. For this reference movie, the image of the center array cam has been selected for further image manipulation. The quality of the depth information is evaluated on simulated data and live action footage. Our experiments show that quality of depth maps depends on array geometry and scene setup

The Potential of Light Fields in Media Productions

“…The main challenges have been the data handling (9,07 GB per second raw data at 41 fps) and the physical setup of 64 cameras with their associated control units. 8x8 cameras is a unique feature of the setup, as recent camera arrays were limited to a 3x3 to 5x5 setup [Ziegler et al 2015] [Sabater et al 2017].…”

“…High quality chroma keying needs significant on set preparation and manual post-processing. Light field data could serve as a powerful new tool [Ziegler et al 2015], not only as image data, but also as flexible and reusable asset. On top it can serve as input for upcoming holographic and AR/VR devices.…”

The Light Field Attachment: Turning a DSLR into a Light Field Camera Using a Low Budget Camera Ring

“…These can first be divided into camera-array methods and single-camera methods. Camera-array methods [15], [16], [17] are capable of capturing light fields with both a high spatial resolution and a large angular range. However, the hardware required for these methods is expensive and complex, is nearly impossible to adapt for portable applications, 1.…”

Modeling depth uncertainty of desynchronized multi-camera systems

“…Using multiple cameras to record video is increasingly relevant for motion capture [1], multi-media production and post-processing [2,3], surveillance and computer vision [4], 3D-TV applications [5], 360-degree video [6], and virtual reality video [7]. In all such applications, accurate depth and 3D information is desirable, as it increases the objective and subjective quality of any depth-dependent composition, rendering and post-processing results.…”

“…This lack of accurate synchronization leads to an imprecise recording and reconstruction of moving element positions in 3D space. The importance of synchronization has been noted [1,2,5], however, this 3D position imprecision -which we call depth uncertainty (∆d) -has not been modeled as a direct consequence of the camera system properties and applications.…”