Deep Synthesis of Cloud Lighting

Abstract

Current appearance models for the sky are able to represent clear sky illumination to a high degree of accuracy. However, these models all lack a common feature of real-skies: clouds. These are an essential component for many applications which rely on realistic skies, such as image editing and synthesis. While clouds can be added to existing sky models through rendering, this is hard to achieve due to the difficulties of representing clouds and the complexities of volumetric light transport. In this work, an alternative approach to this problem is proposed whereby clouds are synthesized using a learned data-driven representation. This leverages a captured collection of High Dynamic Range cloudy sky imagery, and combines this dataset with clear sky models to produce plausible cloud appearance from a coarse representation of cloud positions. This representation is artist controllable, allowing for novel cloudscapes to be rapidly synthesized, and used for lighting virtual environments.

SkyGAN: Realistic Cloud Imagery for Image‐based Lighting

“…Although our dataset also consists of sequential images, we do not exploit the time dependency yet. [SB23] predicts the clouds movement from a single input image. For an image-based relighting approach in neural rendering, Yu et al [YME*20] employ a GAN that fills the image background with realistic sky imagery given a segmentation map.…”

“…We instead build upon initial works [Hoj19,Špa20] that apply GANs to cloud image generation but are missing the direct control over the sun position and the clear sky supervision. Concurrently to our work, a very similar approach has been published [SMDB22]. The authors use a U-Net encoder-decoder architecture to transfer clear sky images of the Hošek-Wilkie sky model together with a cloud segmentation map to realistic cloudy images.…”