Guiding Labelling Effort for Efficient Learning With Georeferenced Images

Abstract

We describe a novel semi-supervised learning method that reduces the labelling effort needed to train convolutional neural networks (CNNs) when processing georeferenced imagery. This allows deep learning CNNs to be trained on a per-dataset basis, which is useful in domains where there is limited learning transferability across datasets. The method identifies representative subsets of images from an unlabelled dataset based on the latent representation of a location guided autoencoder. We assess the method's sensitivities to design options using four different ground-truthed datasets of georeferenced environmental monitoring images, where these include various scenes in aerial and seafloor imagery. Efficiency gains are achieved for all the aerial and seafloor image datasets analysed in our experiments, demonstrating the benefit of the method across application domains. Compared to CNNs of the same architecture trained using conventional transfer and active learning, the method achieves equivalent accuracy with an order of magnitude fewer annotations, and 85 % of the accuracy of CNNs trained conventionally with approximately 10,000 human annotations using just 40 prioritised annotations. The biggest gains in efficiency are seen in datasets with unbalanced class distributions and rare classes that have a relatively small number of observations.

GeoCLR: Georeference Contrastive Learning for Efficient Seafloor Image Interpretation

“…The second k-means clustering avoids selecting similar samples from within each cluster, so that the full variety of images in the dataset can be represented by a small number of annotations. This H-k-means selection was shown to outperform random selection when appropriate latent representations are generated (Yamada et al, 2022). The same work also demonstrated the use of pseudo-labels, generated from the predictions of classical classifiers applied to the latent representations, for CNN fine-tuning, which is also examined in this work.…”

“…However, the method suffers when the number of images in each class is not balanced; since classes are represented in proportion to their relative abundance, those with small populations tend to exhibit poor performance. The hierarchical k-means clustering (Nister and Stewenius, 2006), or H-k-means, method allows for balanced representation of the variety of images present in a dataset without the need for additional human effort, and was shown to be effective for guiding human labeling effort in Yamada et al (2022). In this method, k-means clustering is first applied to latent representations with k = m to find representative clusters of images in the dataset.…”

Implementation of an automated workflow for image-based seafloor classification with examples from manganese-nodule covered seabed areas in the Central Pacific Ocean

“…However, our approach is novel because it allows for the incorporation of domain knowledge through feature space engineering, rather than only relying on similarity in geographic space and spatial auto correlation e.g. as proposed by 52 . Regarding optical image-based seafloor classification, our results revealed seabed substrate classes that had semantic meaning, similar to previous works by 20,21,24,25,53,54 .…”

Leveraging Spatial Metadata in Machine Learning for Improved Objective Quantification of Geological Drill Core

“…In this study we modify two unsupervised learning frameworks originally designed to use 3D geolocational metadata for improved semantic interpretation of seafloor imagery (Yamada et al., 2021; Yamada, Massot‐Campos, et al., 2022; Yamada, Prügel‐Bennett, et al., 2022) to instead use the x‐y coordinate of where an image lies on the surface of a 3D drill core image. The first framework uses an autoencoder that was trained both with and without the addition of this spatial metadata, whereas the second uses two contrastive learning methods, one that makes use of metadata, and another that does not (Table 1).…”

“…To overcome this limitation, Yamada, Prügel‐Bennett, et al. (2022) developed “georeference contrastive learning of visual representation” (GeoCLR) to efficiently train CNNs by leveraging georeferenced metadata. Their data set consisted of 86,772 seafloor images collected by an autonomous underwater vehicle (AUV) from a single locality, and each image had an associated depth, northing and easting.…”