AiRound and CV-BrCT: Novel Multiview Datasets for Scene Classification

Abstract

It is undeniable that aerial/satellite images can provide useful information for a large variety of tasks. But, since these images are always looking from above, some applications can benefit from complementary information provided by other perspective views of the scene, such as ground-level images. Despite a large number of public repositories for both georeferenced photographs and aerial images, there is a lack of benchmark datasets that allow the development of approaches that exploit the benefits and complementarity of aerial/ground imagery. In this paper, we present two new publicly available datasets named AiRound and CV-BrCT. The first one contains triplets of images from the same geographic coordinate with different perspectives of view extracted from various places around the world. Each triplet is composed of an aerial RGB image, a ground-level perspective image, and a Sentinel-2 sample. The second dataset contains pairs of aerial and street-level images extracted from southeast Brazil. We design an extensive set of experiments concerning multi-view scene classification, using early and late fusion. Such experiments were conducted to show that image classification can be enhanced using multi-view data.

Facing the Void: Overcoming Missing Data in Multi-View Imagery

“…Query and top-k images are then processed using networks trained specifically for their domains. Predictions σ for the top-k images are fused using the mean operation [21]:…”

“…Finally, the predictions for the query image (σ f 1 ) and the merged predictions of the top-k scenes (σ f 2 ) are latefused using the product operation [21], thus producing the final classification:…”

“…Two multi-view datasets with very distinct properties were used for the experiments in order to better evaluate the effectiveness of the proposed approach. The first dataset, called AiRound [21], is composed of 11,753 images divided into 11 classes: airport, bridge, church, forest, lake, park, river, skyscraper, stadium, statue, and tower. Some examples of these classes are presented in Figure 3.…”

“…Some examples of these classes are presented in Figure 3. The second one, named CV-BrCT [21], comprises approximately 24k pairs of images unevenly split into 7 urban classes: apartment, house, industrial, parking lot, religious, school, store. Samples of these classes are presented in Figure 4.…”

Scene classification using a combination of aerial and ground images

“…Finally, to calculate the final prediction, we use both softmax scores returned by the inferences, and calculate a fusion of them using the product fusion [22].…”