Evaluating automated benthic fish detection under variable conditions

Abstract

Advances in imaging systems have facilitated the collection of high-volume imagery datasets in fisheries science. To alleviate the costs of sorting these datasets, automated image processing techniques are used. In this study, we investigate a machine learning-enabled imaging technique for automating individual fish detection from stereo image pairs of orange roughy (Hoplostethus atlanticus). We performed a set of object detection experiments to investigate how well a Single Shot Multi-Box Detector (SSD) model worked under dynamic real-world conditions when trained over a small number of epochs. We tested model generalization between the port and starboard side cameras; at variable fish densities; different benthic substrates; and at different altitudes above the seafloor. We show that (1) changes in perspective between starboard and port images are not enough to break the object detector, (2) the object detector begins to have trouble differentiating individuals at high fish densities (>20 fish per image), (3) substrate type does not affect model performance, and (4) altitude is not a major factor contributing to model error. Ideally, this type of real-world dataset exploration should be performed prior to committing the resources to train the final object detector over several hundred epochs.

Demystifying image-based machine learning: a practical guide to automated analysis of field imagery using modern machine learning tools

“…Many of the traditional problems marine scientists currently use imagery to address fall into one of three categories: image classification, object detection, or semantic segmentation. More complex tasks such as tracking (Katija et al, 2021;Irisson et al, 2022), functional trait analysis (Orenstein et al, 2022), pose estimation (Graving et al, 2019), and automated measurements (Fernandes et al, 2020) often rely on these more basic tasks as building blocks.…”

“…A second common task involves detecting and spatially localizing objects of interest within images, a task known as object detection or instance segmentation. Separating instances of the same type of object (e.g., there are nine fish identified as Atlantic cod in this image) in a given image is often crucial if imagery is being used to estimate abundances (Moeller et al, 2018), and most object detection pipelines can also be trained to detect objects of many different classes, which is valuable for analyzing images that contain multiple objects of interest that belong to different classes (see Scoulding et al, 2022 for a discussion of limitations at high density).…”

Demystifying image-based machine learning: A practical guide to automated analysis of field imagery using modern machine learning tools

“…A distribution shift occurs when the imagery on which an ML pipeline is trained differs in some systematic way from the imagery on which the pipeline is deployedthat is, the new imagery the ML method is being used to analyze. The term "distribution shift" refers to a generic set of differences that may occur between one set of images (the "in-domain" set) and another (the "out-of-domain" set), including things like differences in lighting, camera attributes, image scene statistics, background clutter, turbidity, and the relative abundances and appearances of different classes of objects (Taori et al 2020, Scoulding et al 2022, Wyatt et al 2022. This problem arises often in scientific sampling, because in this setting, the goal is often to extract data from one image set collected at a particular place and time, and to compare it to data from another image data set collected at some other place or time.…”

“…Separating instances of the same type of object (e.g., there are nine fish identified as Atlantic cod in this image) in a given image is often crucial if imagery is being used to estimate abundances (Moeller et al . 2018, but see Scoulding et al . 2022 for a discussion of limitations at high density), and most object detection pipelines can be trained to detect objects of many different classes, which is valuable for analyzing images that contain multiple objects of interest of different types (e.g., many distinct phytoplankton species or morphotypes in the same image, Irisson et al .…”

“…The term “distribution shift” refers to a generic set of differences that may occur between one set of images (the “in-domain” set) and another (the “out-of-domain” set), including things like differences in lighting, camera attributes, image scene statistics, background clutter, turbidity, and the relative abundances and appearances of different classes of objects (Taori et al . 2020, Scoulding et al . 2022, Wyatt et al .…”