Deep Learning for 3d Building Reconstruction: A Review

Abstract

Abstract. 3D building reconstruction using Earth Observation (EO) data (aerial and satellite imagery, point clouds, etc.) is an important and active research topic in different fields, such as photogrammetry, remote sensing, computer vision and Geographic Information Systems (GIS). Nowadays 3D city models have become an essential part of 3D GIS environments and they can be used in many applications and analyses in urban areas. The conventional 3D building reconstruction methods depend heavily on the data quality and source; and manual efforts are still needed for generating the object models. Several tasks in photogrammetry and remote sensing have been revolutionized by using deep learning (DL) methods, such as image segmentation, classification, and 3D reconstruction. In this study, we provide a review on the state-of-the-art machine learning and in particular the DL methods for 3D building reconstruction for the purpose of city modelling using EO data. This is the first review with a focus on object model generation based on the DL methods and EO data. A brief overview of the recent building reconstruction studies with DL is also given. We have investigated the different DL architectures, such as convolutional neural networks (CNNs), generative adversarial networks (GANs), and the combinations of conventional approaches with DL in this paper and reported their advantages and disadvantages. An outlook on the future developments of 3D building modelling based on DL is also presented.

Deep Learning for 3D Reconstruction, Augmentation, and Registration: A Review Paper

“…Typically, data-driven techniques lack robustness and are extremely susceptible to data noise. Because data-driven methods are sensitive to noise, pre-processing data is a crucial step in preventing inaccurate outcomes [ 43 ].…”

“…With the recent rapid growth in 3D building models and the availability of a wide variety of 3D shapes, DL-based 3D reconstruction has become increasingly practical. It is possible to train DL models to recognise 3D shapes and all of their attributes [ 43 ].…”

“…Second, they rely heavily on the calibre and content of the images in addition to a properly calibrated camera. By employing deep networks to automatically learn 3D shape semantics from pictures or point clouds, DL-based 3D reconstruction techniques go beyond these obstacles [ 43 , 58 ].…”

Hybrid Method of Connection Evaluation and Framework Optimization for Building Surface Reconstruction

“…Data-driven methods and model-driven methods are the two main kinds of building modeling methods [10]. Between them, the data-driven method employs a bottom-up approach that begins with data extraction to reconstruct the geometric model [11,12].…”

Machine-Learning-Enhanced Procedural Modeling for 4D Historical Cities Reconstruction

“…Meanwhile, at an architectural scale, we may resort to educated guesses; one way to solve this contradictory objective in large-scale scenarios is to use statistical machine learning methods to infer the most probable values in an objective way, based on available data. While deep learning techniques are well suited to process remote sensing data [6,7], already vectorized GIS datasets represent an easier case that can effectively be tackled by classical machine learning algorithms. Within this category, we favor the use of ensemble methods over regression or nearest neighbor algorithms, because of their ability to deal with missing values in input parameters.…”