Fast and Regularized Reconstruction of Building Façades From Street-View Images Using Binary Integer Programming

Abstract

Abstract. Regularized arrangement of primitives on building façades to aligned locations and consistent sizes is important towards structured reconstruction of urban environment. Mixed integer linear programing was used to solve the problem, however, it is extremely time consuming even for state-of-the-art commercial solvers. Aiming to alleviate this issue, we cast the problem into binary integer programming, which omits the requirements for real value parameters and is more efficient to be solved. Firstly, the bounding boxes of the primitives are detected using the YOLOv3 architecture in real-time. Secondly, the coordinates of the upper left corners and the sizes of the bounding boxes are automatically clustered in a binary integer programming optimization, which jointly considers the geometric fitness, regularity and additional constraints; this step does not require a priori knowledge, such as the number of clusters or pre-defined grammars. Finally, the regularized bounding boxes can be directly used to guide the façade reconstruction in an interactive environment. Experimental evaluations have revealed that the accuracies for the extraction of primitives are above 0.82, which is sufficient for the following 3D reconstruction. The proposed approach only takes about 10% to 20% of the runtime than previous approach and reduces the diversity of the bounding boxes to about 20% to 50%.

Semi-Supervised Adversarial Recognition of Refined Window Structures for Inverse Procedural Façade Modeling

“…In addition, a set of suitable window bounding boxes {B i } is marked as the same cluster, and a single component instance C k is created for a window cluster. Automatic clustering is possible, e.g., according to the geometry information Hu et al (2020); however, we find the accuracy of automatic clustering is not good enough, and interactive processing for this step is also quite efficient.…”

“…Therefore, we assume the details of windows are visible and apparent in the texture image. The bounding boxes of windows can be detected using a standard two-stage object detection method (Ren et al, 2015) at very high confidence (Figure 2c), as in our previous work (Hu et al, 2020); more sophisticated approaches for façade parsing are also possible (Liu et al, 2020;Ma et al, 2020). In addition, we also allow interactive sketching of the window regions in cases of occlusion.…”

“…Two typical strategies in image processing are widely adopted, e.g., object detection (Ren et al, 2015) and semantic segmentation (Long et al, 2015). For the former, the rectangular regions of the windows could be detected using a typical detector (Ren et al, 2015) and used for façade reconstruction (Hu et al, 2020;Hensel et al, 2019;Kong and Fan, 2020); however, there is still ample space to improve the localization accuracy, e.g., the intersection of union (IoU). For the latter, pixel-wise segmentation results can be learned end-to-end (Mathias et al, 2016;Gadde et al, 2018); fusing with point clouds (Gadde et al, 2018) and multi-view voting (Ma et al, 2020) can also be adopted to improve the segmentation results.…”

“…A straightforward approach for regularization explicitly offsets adjacent points or boundaries in a least-squares optimization (Arikan et al, 2013;Zhu et al, 2020b;Xie et al, 2018). Because least-squares optimization cannot model the logic operators, such as sameColumn and sameWidth (Dehbi et al, 2017), clustering (Fan et al, 2021), learned Markov Logic Networks (Dehbi et al, 2017), or binary integer programming (Hu et al, 2020;Hensel et al, 2019;Monszpart et al, 2015) are exploited to snap the feature points on the façade.…”

“…With the advent of aerial oblique images and ground mobile mapping systems, city-scale photo-realistic 3D models with façade visibility are readily available (Verdie et al, 2015;Han et al, 2021). Detailed analysis and parsing of façade windows (Fan et al, 2021;Hu et al, 2020), which enriches the 3D models with semantic information conformal to the LOD-3 (Level-of-Details) protocol in CityGML (Gröger and Plümer, 2012), have recently raised extensive attention in the community. Although the LOD-3 models already contain computational information for various applications, the taxonomy and refined structure of windows are often ignored (Zhu et al, 2020a).…”