DeepFacade: A Deep Learning Approach to Facade Parsing With Symmetric Loss

Liu, Hantang; Xu, Yinghao; Zhang, Jialiang; Zhu, Jianke; Yang, Li; Hoi, Steven C. H.

doi:10.1109/tmm.2020.2971431

Cited by 38 publications

(48 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To compare our method with other window extraction approaches [22][23][24][25]27], we retrained and evaluated the proposed method on several datasets: eTRIMS, ECP, CMP, Graz50, and ParisArtDeco. The pixel accuracy is used as a metric in these previous studies, which can be calculated through Equation (7).…”

Section: Comparisons With Other Window Extraction Methodsmentioning

confidence: 99%

“…Moreover, they proposed a symmetric loss term to improve the results. Recently, the authors introduced a Region Proposal Network (RPN) into their symmetric loss term [25]. The distances between the clustered windows and the detected bounding boxes are treated as a loss metric.…”

Section: Related Workmentioning

confidence: 99%

“…Unlike traditional methods, deep learning methods can deal with facades without strict structures. A number of CNN-based approaches have been proposed for facade segmentation [22][23][24][25][26] and window detection [27,28]. However, these methods only regard each window as an individual component.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

DeepWindows: Windows Instance Segmentation through an Improved Mask R-CNN Using Spatial Attention and Relation Modules

Sun

Malihi

et al. 2022

IJGI

View full text Add to dashboard Cite

Windows, as key components of building facades, have received increasing attention in facade parsing. Convolutional neural networks have shown promising results in window extraction. Most existing methods segment a facade into semantic categories and subsequently employ regularization based on the structure of manmade architectures. These methods merely concern the optimization of individual windows, without considering the spatial areas or relationships of windows. This paper presents a novel windows instance segmentation method based on Mask R-CNN architecture. The method features a spatial attention region proposal network and a relation module-enhanced head network. First, an attention module is introduced in the region proposal network to generate a spatial attention map, then the attention map is multiplied with the objectness scores of the classification branch. Second, for the head network, relation modules are added to model the spatial relationships between proposals. Appearance and geometric features are combined for instance recognition. Furthermore, we constructed a new window instance segmentation dataset with 1200 annotated images. With our dataset, the average precisions of our method on detection and segmentation increased from 53.1% and 53.7% to 56.4% and 56.7% compared with Mask R-CNN. A comparison with state-of-the-art methods also proves the predominance of our proposed method.

show abstract

Section: Comparisons With Other Window Extraction Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

DeepWindows: Windows Instance Segmentation through an Improved Mask R-CNN Using Spatial Attention and Relation Modules

Sun

Malihi

et al. 2022

IJGI

View full text Add to dashboard Cite

show abstract

“…For the latter, pixel-wise segmentation results can be learned end-to-end (Mathias et al, 2016;Gadde et al, 2018); fusing with point clouds (Gadde et al, 2018) and multi-view voting (Ma et al, 2020) can also be adopted to improve the segmentation results. In addition, the combination of the two strategies (termed instance segmentation) (He et al, 2017), which detects the bounding boxes first and conducts pixel-wise segmentation inside each object, is also a practical approach (Liu et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

Semi-Supervised Adversarial Recognition of Refined Window Structures for Inverse Procedural Façade Modeling

Huang¹,

Liang²,

Ding³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep learning methods are notoriously data-hungry, which requires a large number of labeled samples. Unfortunately, the large amount of interactive sample labeling efforts has dramatically hindered the application of deep learning methods, especially for 3D modeling tasks, which require heterogeneous samples. To alleviate the work of data annotation for learned 3D modeling of façades, this paper proposed a semi-supervised adversarial recognition strategy embedded in inverse procedural modeling. Beginning with textured LOD-2 (Level-of-Details) models, we use the classical convolutional neural networks to recognize the types and estimate the parameters of windows from image patches. The window types and parameters are then assembled into procedural grammar. A simple procedural engine is built inside an existing 3D modeling software, producing fine-grained window geometries. To obtain a useful model from a few labeled samples, we leverage the generative adversarial network to train the feature extractor in a semi-supervised manner. The adversarial training strategy can also exploit unlabeled data to make the training phase more stable. Experiments using publicly available façade image datasets reveal that the proposed training strategy can obtain about 10% improvement in classification accuracy and 50% improvement in parameter estimation under the same network structure. In addition, performance gains are more pronounced when testing against unseen data featuring different façade styles.

show abstract

“…For application in object detection, it can detect cracks by YOLOv2 which is object detection network structure [15]. For application in semantic segmentation, it can segment many façade objects efficiently [16,17]. Hyperspectral image classification also can be used [37,38] In this study, we used object detection method to detect object-type contaminants.…”

Section: Object-type Contaminant Detectionmentioning

confidence: 99%

Contaminated Facade Identification Using Convolutional Neural Network and Image Processing

et al. 2020

View full text Add to dashboard Cite

In recent years, as number of new building getting larger, there has been an increased interest in the cleaning of exterior walls. Accordingly, there is a growing interest in automatic cleaning robots that move around the outer building façade. These robots are also required to apply different cleaning methods to remove various contaminants on the outer wall of the building. However, current surface contaminant detection systems can either detect only a single type of contaminant, or are not compact enough for installation on mobile platforms that move around the outer façade. As cleaning workers are able to distinguish various contaminants with the naked eye, we aim to solve this problem by developing a machinevision system using convolutional neural networks (CNNs) and image processing methods. As it is a compact system that uses only a camera to take pictures and a processor to process the images, it is suitable for applications involving mobile platforms. Object-type contaminants such as avian feces are handled by the YOLOv3 module using the object-detection algorithm. Area-type contaminants such as rusty stains are processed using the color-detection module using the HSV color space, median filter, and flood fill algorithm. Particle-type contaminants such as dust are handled by the grayscale module, converting images to grayscale images and then comparing the average brightness with a reference that is provided in advance. This proposed machine vision system will detect objects, areas, and particle-type contaminants with a single image and some reference images provided in advance.

show abstract

DeepFacade: A Deep Learning Approach to Facade Parsing With Symmetric Loss

Cited by 38 publications

References 42 publications

DeepWindows: Windows Instance Segmentation through an Improved Mask R-CNN Using Spatial Attention and Relation Modules

DeepWindows: Windows Instance Segmentation through an Improved Mask R-CNN Using Spatial Attention and Relation Modules

Semi-Supervised Adversarial Recognition of Refined Window Structures for Inverse Procedural Façade Modeling

Contaminated Facade Identification Using Convolutional Neural Network and Image Processing

Contact Info

Product

Resources

About