Bounding Boxes Are All We Need: Street View Image Classification via Context Encoding of Detected Buildings

Abstract

Street view images classification aiming at urban land use analysis is difficult because the class labels (e.g., commercial area), are concepts with higher abstract level compared to the ones of general visual tasks (e.g., persons and cars). Therefore, classification models using only visual features often fail to achieve satisfactory performance. In this paper, a novel approach based on a "Detector-Encoder-Classifier" framework is proposed. Instead of using visual features of the whole image directly as common image-level models based on convolutional neural networks (CNNs) do, the proposed framework firstly obtains the bounding boxes of buildings in street view images from a detector. Their contextual information such as the co-occurrence patterns of building classes and their layout are then encoded into metadata by the proposed algorithm "CODING" (Context encOding of Detected buildINGs). Finally, these bounding box metadata are classified by a recurrent neural network (RNN). In addition, we made a dual-labeled dataset named "BEAUTY" (Building dEtection And Urban funcTional-zone portraYing) of 19,070 street view images and 38,857 buildings based on the existing BIC GSV [1]. The dataset can be used not only for street view image classification, but also for multi-class building detection. Experiments on "BEAUTY" show that the proposed approach achieves a 12.65% performance improvement on macro-precision and 12% on macro-recall over image-level CNN based models. Our code and dataset are available at https: //github.com/kyle-one/Context-Encoding-of-Detected-Buildings/ Index Terms-Street view images classification, context encoding, building detection, urban land use classification, urban functional zone, RNN.

Urban land-use analysis using proximate sensing imagery: a survey

“…Their experiments found a strong correlation between the object counts in social media images and building functions. Zhao et al (2020) proposed a 'Detector-Encoder-Classifier' network to firstly detect the building of different categories in GSV images using state-of-the-art object detectors (Ren et al 2015), (Cai and Vasconcelos 2018). Then the detected bounding boxes metadata is sent into the Recurrent Neural Network (RNN) to conduct urban land-use classification.…”