Leveraging Metadata in Representation Learning With Georeferenced Seafloor Imagery

Yamada, Takaki; Massot-Campos, Miquel; Prügel-Bennett, Adam; Williams, Stefan B.; Pizarro, Oscar; Thornton, Blair

doi:10.1109/lra.2021.3101881

Cited by 11 publications

(3 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These challenges are inherent in ocean sampling and limit the ability of fully autonomous systems to adjust their behavior based on visual signals. There are several bleedingedge, pure ML solutions that are well-worth experimentation: Open World Object Detection frameworks to identify novel classes in a new domain (Joseph et al, 2021); contrastive learning to identify out-of-distribution samples and study areas (Yamada et al, 2021); and uncertainty quantification to compute robust confidence thresholds around ML outputs for hypothesis testing (Angelopoulos et al, 2022). Additionally, the promise of reinforcement learning holds potential for addressing the control problem associated with handling more complex animal behavior (e.g., swimming): an area where the current implementation of simple PID thruster-effort-based control struggles.…”

Section: Discussionmentioning

confidence: 99%

DeepSTARia: enabling autonomous, targeted observations of ocean life in the deep sea

Barnard,

Daniels,

Roberts

et al. 2024

Front. Mar. Sci.

View full text Add to dashboard Cite

The ocean remains one of the least explored places on our planet, containing myriad life that are either unknown to science or poorly understood. Given the technological challenges and limited resources available for exploring this vast space, more targeted approaches are required to scale spatiotemporal observations and monitoring of ocean life. The promise of autonomous underwater vehicles to fulfill these needs has largely been hindered by their inability to adapt their behavior in real-time based on what they are observing. To overcome this challenge, we developed Deep Search and Tracking Autonomously with Robotics (DeepSTARia), a class of tracking-by-detection algorithms that integrate machine learning models with imaging and vehicle controllers to enable autonomous underwater vehicles to make targeted visual observations of ocean life. We show that these algorithms enable new, scalable sampling strategies that build on traditional operational modes, permitting more detailed (e.g., sharper imagery, temporal resolution) autonomous observations of underwater concepts without supervision and robust long-duration object tracking to observe animal behavior. This integration is critical to scale undersea exploration and represents a significant advance toward more intelligent approaches to understanding the ocean and its inhabitants.

show abstract

Section: Discussionmentioning

confidence: 99%

DeepSTARia: enabling autonomous, targeted observations of ocean life in the deep sea

Barnard,

Daniels,

Roberts

et al. 2024

Front. Mar. Sci.

View full text Add to dashboard Cite

show abstract

“…Cai et al (2022c) proposed an enhanced dilated convolution framework for underwater blurred target recognition. Yamada et al (2021) proposed a novel selfsupervised representation learning method. The method allows deep learning convolutional autoencoders to utilize multiple metadata sourced to normalize their learning.…”

Section: Image Object Recognitionmentioning

confidence: 99%

Multi-scale aware turbulence network for underwater object recognition

Zhou,

Cai,

Jia

et al. 2024

Front. Mar. Sci.

View full text Add to dashboard Cite

Underwater imagery is subject to distortion, and the presence of turbulence in the fluid medium poses difficulties in accurately discerning objects. To tackle these challenges pertaining to feature extraction, this research paper presents a novel approach called the multi-scale aware turbulence network (MATNet) method for underwater object identification. More specifically, the paper introduces a module known as the multi-scale feature extraction pyramid network module, which incorporates dense linking strategies and position learning strategies to preprocess object contour features and texture features. This module facilitates the efficient extraction of multi-scale features, thereby enhancing the effectiveness of the identification process. Following that, the extracted features undergo refinement through comparison with positive and negative samples. Ultimately, the study introduces multi-scale object recognition techniques and establishes a multi-scale object recognition network for the precise identification of underwater objects, utilizing the enhanced multi-scale features. This process entails rectifying the distorted image and subsequently recognizing the rectified object. Extensive experiments conducted on an underwater distorted image enhancement dataset demonstrate that the proposed method surpasses state-of-the-art approaches in both qualitative and quantitative evaluations.

show abstract

“…The method significantly outperformed standard convolutional autoencoders without location regularization, achieving a factor of 2 improvement in normalized mutual information when applied to clustering and content-based retrieval tasks. In Yamada et al (2021a), the LGA is extended to utilize other types of metadata, such as depth information, where the continuity in measurements has potential correlation with image appearance, where it was demonstrated that these terms can be included without risk of performance degradation through the design of a robust regularization process.…”

Section: Self-supervised Learning For Seafloor Imagerymentioning

confidence: 99%

GeoCLR: Georeference Contrastive Learning for Efficient Seafloor Image Interpretation

Yamada¹,

Prügel-Bennett²,

Williams³

et al. 2022

Self Cite

View full text Add to dashboard Cite

This paper describes georeference contrastive learning of visual representation (GeoCLR) for efficient training of deep-learning convolutional neural networks (CNNs). The method leverages georeference information by generating a similar image pair using images taken of nearby locations, and contrasting these with an image pair that is far apart. The underlying assumption is that images gathered within a close distance are more likely to have similar visual appearance, where this can be reasonably satisfied in seafloor robotic imaging applications where image footprints are limited to edge lengths of a few meters and are taken so that they overlap along a vehicle’s trajectory, whereas seafloor substrates and habitats have patch sizes that are far larger. A key advantage of this method is that it is self-supervised and does not require any human input for CNN training. The method is computationally efficient, where results can be generated between dives during multi-day autonomous underwater vehicle (AUV) missions using computational resources that would be accessible during most oceanic field trials. We apply GeoCLR to habitat classification on a dataset that consists of ~86,000 images gathered using an AUV. We demonstrate how the latent representations generated by GeoCLR can be used to efficiently guide human annotation efforts, where the semi-supervised framework improves classification accuracy by an average of 10.2% compared to the state-of-the-art SimCLR using the same CNN and equivalent number of human annotations for training.

show abstract

Leveraging Metadata in Representation Learning With Georeferenced Seafloor Imagery

Cited by 11 publications

References 23 publications

DeepSTARia: enabling autonomous, targeted observations of ocean life in the deep sea

DeepSTARia: enabling autonomous, targeted observations of ocean life in the deep sea

Multi-scale aware turbulence network for underwater object recognition

GeoCLR: Georeference Contrastive Learning for Efficient Seafloor Image Interpretation

Contact Info

Product

Resources

About