DeepWindows: Windows Instance Segmentation through an Improved Mask R-CNN Using Spatial Attention and Relation Modules

Abstract

Windows, as key components of building facades, have received increasing attention in facade parsing. Convolutional neural networks have shown promising results in window extraction. Most existing methods segment a facade into semantic categories and subsequently employ regularization based on the structure of manmade architectures. These methods merely concern the optimization of individual windows, without considering the spatial areas or relationships of windows. This paper presents a novel windows instance segmentation method based on Mask R-CNN architecture. The method features a spatial attention region proposal network and a relation module-enhanced head network. First, an attention module is introduced in the region proposal network to generate a spatial attention map, then the attention map is multiplied with the objectness scores of the classification branch. Second, for the head network, relation modules are added to model the spatial relationships between proposals. Appearance and geometric features are combined for instance recognition. Furthermore, we constructed a new window instance segmentation dataset with 1200 annotated images. With our dataset, the average precisions of our method on detection and segmentation increased from 53.1% and 53.7% to 56.4% and 56.7% compared with Mask R-CNN. A comparison with state-of-the-art methods also proves the predominance of our proposed method.

LCA Calculation of Retrofitting Scenarios Using Geometric Model Reconstruction and Semantic Enrichment of Point Clouds and Images

“…Results of instance segmentation and model enrichment of windows. As shown in Figure 2, instance segmentation of windows using the DeepWindows network by (Sun et al 2022) provide sufficient results for both types of input data, Google Street View images as well as Smartphone images. For the further enrichment of the geometric model, we are manually assigning the detected windows in the images to the related façade surfaces.…”

“…As we need information on the number of windows and the surface, we use instance segmentation. Sun et al trained a Mask RNN called "DeepWindows," using Detectron2 (Sun et al 2022), which we used for identifying the windows. Nevertheless, the assignment of windows to the reconstructed 3D surface is here done manually, however, as a temporary solution only.…”

“…Although they only focus on windows, their approach using instance segmentation produces a bounding box and segmentation mask for each window instance. More recently, Sun et al also proposed an improved Mask R-CNN network using spatial attention and relation modules for windows instance segmentation, called "DeepWindows" (Sun et al 2022).…”

Cross Field-Based Segmentation and Learning-Based Vectorization for Rectangular Windows

“…We compare the segmentation accuracy of our segmentation model (denoted as Ours ) with other state-of-the-art methods, including the DeepFacade network [3] (denoted as DeepFacade), the refined DAN-PSPNet with symmetric loss function [28] (denoted as DAN-PSPNet-L sym ), the Frame Field Polygonization network [5] (denoted as FFP) and the DeepWindows network [74] (denoted as DeepWindows). Among them, we use the source code provided by the authors to implement the FFP [5] and the DeepWindows [74] networks. We cannot reproduce results of the DeepFacade [3] and PSPNet [28] networks, as their codes are either not open-source or written in an outdated deep learning framework.…”