DepthGAN: GAN-based Depth Generation of Indoor Scenes from Semantic Layouts

Abstract

Limited by the computational efficiency and accuracy, generating complex 3D scenes remains a challenging problem for existing generation networks. In this work, we propose DepthGAN, a novel method of generating depth maps with only semantic layouts as input. First, we introduce a well-designed cascade of transformer blocks as our generator to capture the structural correlations in depth maps, which makes a balance between global feature aggregation and local attention. Meanwhile, we propose a cross-attention fusion module to guide edge preservation efficiently in depth generation, which exploits additional appearance supervision information. Finally, we conduct extensive experiments on the perspective views of the Structured3d panorama dataset and demonstrate that our DepthGAN achieves superior performance both on quantitative results and visual effects in the depth generation task. Furthermore, 3D indoor scenes can be reconstructed by our generated depth maps with reasonable structure and spatial coherency.

BallGAN: 3D-aware Image Synthesis with a Spherical Background

“…Although the rendered images are realistic, the image synthesis process conceal the degenerate solutions for the underlying 3D geometries, e.g., broken shapes and faces attached to walls as shown in Figure 2 do not resemble the foreground geometry shown in the resulting images. Some methods try to circumvent the degenerate solutions by involving external guides such as prior knowledge [38] or off-the-shelf tools [18].…”