Self-supervised Outdoor Scene Relighting

Abstract

Outdoor scene relighting is a challenging problem that requires good understanding of the scene geometry, illumination and albedo. Current techniques are completely supervised, requiring high quality synthetic renderings to train a solution. Such renderings are synthesized using priors learned from limited data. In contrast, we propose a self-supervised approach for relighting. Our approach is trained only on corpora of images collected from the internet without any user-supervision. This virtually endless source of training data allows training a general relighting solution. Our approach first decomposes an image into its albedo, geometry and illumination. A novel relighting is then produced by modifying the illumination parameters. Our solution capture shadow using a dedicated shadow prediction map, and does not rely on accurate geometry estimation. We evaluate our technique subjectively and objectively using a new dataset with ground-truth relighting. Results show the ability of our technique to produce photo-realistic and physically plausible results, that generalizes to unseen scenes.

NeurSF: Neural Shading Field for Image Harmonization

“…Still, the illumination estimation is only considered for specific objects rather than natural scenes. Given multi-view images, Yu et al [42] proposed the first single imagebased outdoor scene relighting method along with lighting estimation for the scene. They used the spherical harmonics lighting [28] to generate the shading and it could not handle cast shadows caused by occlusion.…”

“…They used the spherical harmonics lighting [28] to generate the shading and it could not handle cast shadows caused by occlusion. Moreover, it is worth noting that although these relighting methods [19,20,31,34,38,42,46] with illumination estimation can be applied to image harmonization, additional computational overhead would be introduced, since illumination estimation for the background image is often accompanied by the estimation of other physical attributes in the background image. In other words, these relighting methods are not specifically designed for image harmonization.…”

De-rendering the World's Revolutionary Artefacts

“…However, the complexity in acquiring controlled multiple images of the same real-world object has led these models to be trained again only on synthetic data. Some recent works leverage photo collections of real scenes [22,48,47,27], but are often restricted to famous landmarks or street view imagery. Learning from unannotated image collections.…”

Towers of Babel: Combining Images, Language, and 3D Geometry for Learning Multimodal Vision

“…Diverse photo collections of landmarks are unified by the underlying 3D scene geometry, despite the fact that a scene can look dramatically different from one image to the next due to varying illumination, alternating seasons, or special events. This geometric anchoring can be exploited when learning a range of geometry-related vision tasks, such as novel view synthesis [35,29], singleview depth prediction [28], and relighting [60,59], that require large amounts of diverse training data. However, prior work on tourist photos of landmarks has focused almost exclusively on lower-level reconstruction tasks, and not on higher-level scene understanding or recognition tasks.…”