Geo-semantic segmentation

Abstract

The availability of GIS (Geographical Information System) databases for many urban areas, provides a valuable source of information for improving the performance of many computer vision tasks. In this paper, we propose a method which leverages information acquired from GIS databases to perform semantic segmentation of the image alongside with geo-referencing each semantic segment with its address and geo-location. First, the image is segmented into a set of initial super-pixels. Then, by projecting the information from GIS databases, a set of priors are obtained about the approximate location of the semantic entities such as buildings and streets in the image plane. However, there are significant inaccuracies (misalignments) in the projections, mainly due to inaccurate GPS-tags and camera parameters. In order to address this misalignment issue, we perform data fusion such that it improves the segmentation and GIS projections accuracy simultaneously with an iterative approach. At each iteration, the projections are evaluated and weighted in terms of reliability, and then fused with the super-pixel segmentations. First segmentation is performed using random walks, based on the GIS projections. Then the global transformation which best aligns the projections to their corresponding semantic entities is computed and applied to the projections to further align them to the content of the image. The iterative approach continues until the projections and segments are well aligned.

The Cityscapes Dataset for Semantic Urban Scene Understanding

“…caravan 1,2 Vehicles that (appear to) contain living quarters. This also includes trailers that are used for living and has priority over the trailer class.…”

“…trailer 1,2 Includes trailers that can be attached to any vehicle, but excludes trailers attached to trucks. The latter are included in the truck label.…”

“…1 Single instance annotation available. 2 Not included in challenges. This includes areas of the image that are difficult to identify/label due to occlusion/distance, as well as non-movable objects that do not match any of the non-void categories, e.g.…”

“…1 Single instance annotation available. 2 Not included in challenges. GT segmentation with [41] Table 9.…”

Beyond Pixels: Image Provenance Analysis Leveraging Metadata

“…Image metadata is a valuable asset for improving results for vision-based problems such as image retrieval [61], semantic segmentation [5], and manipulation detection [34]. Our work demonstrates that the task of image provenance analysis also benefits from metadata.…”