Single Image Façade Segmentation and Computational Rephotography of House Images Using Deep Learning

Abstract

Rephotography is the process of recapturing the photograph of a location from the same perspective in which it was captured earlier. A rephotographed image is the best presentation to visualize and study the social changes of a location over time. Traditionally, only expert artists and photographers are capable of generating the rephotograph of any specific location. Manual editing or human eye judgment that is considered for generating rephotographs normally requires a lot of precision, effort and is not always accurate. In the era of computer science and deep learning, computer vision techniques make it easier and faster to perform precise operations to an image. Until now many research methodologies have been proposed for rephotography but none of them is fully automatic. Some of these techniques require manual input by the user or need multiple images of the same location with 3D point cloud data while others are only suggestions to the user to perform rephotography. In historical records/archives most of the time we can find only one 2D image of a certain location. Computational rephotography is a challenge in the case of using only one image of a location captured at different timestamps because it is difficult to find the accurate perspective of a single 2D historical image. Moreover, in the case of building rephotography, it is required to maintain the alignments and regular shape. The features of a building may change over time and in most of the cases, it is not possible to use a features detection algorithm to detect the key features. In this research paper, we propose a methodology to rephotograph house images by combining deep learning and traditional computer vision techniques. The purpose of this research is to rephotograph an image of the past based on a single image. This research will be helpful not only for computer scientists but also for history and cultural heritage research scholars to study the social changes of a location during a specific time period, and it will allow users to go back in time to see how a specific place looked in the past. We have achieved good, fully automatic rephotographed results based on façade segmentation using only a single image.

Computer vision and machine learning approaches for metadata enrichment to improve searchability of historical newspaper collections

“…Textual information can be visualized using word clouds (Koivunen-Niemi and Masoodian, 2020;. Illustrations of buildings can be used for rephotography (Ali et al, 2021) of historical locations, while location information can be used for geo-localization (Theiner et al, 2021).…”

Understanding Urban Residents’ Walking Exercise Preferences: An Empirical Study Using Street View Images and Trajectory Data

“…DeepLabv3 tackles this challenge by employing atrous convolutions and atrous spatial pyramid pooling (ASPP) modules, which have improved the mIOU from 34.0 to 44.1 on the ADE20K dataset compared to its predecessor, DeepLabv2. Due to its superior performance and satisfactory results, DeepLabV3 has been widely used for street view image segmentation [52][53][54][55][56]. We implemented this model using the Gluon package [57].…”

Digital 3D Technologies for Humanities Research and Education: An Overview

“…Currently, these technologies can only recognize well documented and visually distinctive landmark buildings [464,746], but fail to deal with less distinctive architecture, e.g., houses of similar style. To overcome the problem of very small training datasets, single or even zero shot learning have been tested for cultural heritage [747,748] to enhance the quality of models trained on sparse data. In another approach, single photo 3D reconstruction [466] has been successfully applied to retrieve 3D information from single images.…”