Machine learning applied to big data from marine cabled observatories: A case study of sablefish monitoring in the NE Pacific

Abstract

Ocean observatories collect large volumes of video data, with some data archives now spanning well over a few decades, and bringing the challenges of analytical capacity beyond conventional processing tools. The analysis of such vast and complex datasets can only be achieved with appropriate machine learning and Artificial Intelligence (AI) tools. The implementation of AI monitoring programs for animal tracking and classification becomes necessary in the particular case of deep-sea cabled observatories, as those operated by Ocean Networks Canada (ONC), where Petabytes of data are now collected each and every year since their installation. Here, we present a machine-learning and computer vision automated pipeline to detect and count sablefish (Anoplopoma fimbria), a key commercially exploited species in the N-NE Pacific. We used 651 hours of video footage obtained from three long-term monitoring sites in the NEPTUNE cabled observatory, in Barkley Canyon, on the nearby slope, and at depths ranging from 420 to 985 m. Our proposed AI sablefish detection and classification pipeline was tested and validated for an initial 4.5 month period (Sep 18 2019-Jan 2 2020), and was a first step towards validation for future processing of the now decade-long video archives from Barkley Canyon. For the validation period, we trained a YOLO neural network on 2917 manually annotated frames containing sablefish images to obtain an automatic detector with a 92% Average Precision (AP) on 730 test images, and a 5-fold cross-validation AP of 93% (± 3.7%). We then ran the detector on all video material (i.e., 651 hours from a 4.5 month period), to automatically detect and annotate sablefish. We finally applied a tracking algorithm on detection results, to approximate counts of individual fishes moving on scene and obtain a time series of proxy sablefish abundance. Those proxy abundance estimates are among the first to be made using such a large volume of video data from deep-sea settings. We discuss our AI results for application on a decade-long video monitoring program, and particularly with potential for complementing fisheries management practices of a commercially important species.

Automatic detection and classification of coastal Mediterranean fish from underwater images: Good practices for robust training

“…Using YOLO, we addressed specific areas that required further investigation, particularly the "domain shift" phenomenon (Kalogeiton et al, 2016;Ditria et al, 2020) characterized by a decrease in classification performance with varying habitat backgrounds and fish species assemblages. Automatic fish classification often involves the use of relative or absolute (e.g., Campos-Candela et al, 2018) abundance estimators that utilize underwater baited cameras (Connolly et al, 2021) or cabled observatories (Bonofiglio et al, 2022) to count, classify or track fish (Saleh et al, 2022). These underwater images differ significantly from typical free datasets that contain single individuals; these images contain a high diversity of species and large variability in abundance, resulting in reduced classification success.…”

Artificial intelligence for fish behavior recognition may unlock fishing gear selectivity

“…An algorithm is trained to identify features of fishes and localize regions in a scene. The YOLO (You Only Look Once; Redmon et al, 2016) object detection framework has been frequently used for fish detection and species classification on 2D images (Cai et al, 2020;Jalal et al, 2020;McIntosh et al, 2020;Raza and Hong, 2020;Bonofiglio et al, 2022;Knausgård et al, 2021). The YOLO algorithm and its different versions are widely used since its detecting speed on an entire image are faster and more accurate than classic object detectors (for technical specifications, see: Redmon et al, 2016).…”

“…For temperate fishes, only a few commercial species can be automatically identified by existing models but are nonetheless gaining more recognition. Bonofiglio et al (2022) trained an AI pipeline to detect and track sablefish, Anoplopoma fimbria, in an underwater canyon in North America on ~650 hours of video recording with ~9000 manual annotations. Due to growing fish databases and application of image processing techniques, AI models can now detect fishes with human-like accuracy in some species such as Scythe butterfly fish (Benson et al, 2013), some tropical species (Spampinato et al, 2010), and mesopelagic species (Allken et al, 2021a).…”

A Deep-Learning Based Pipeline for Estimating the Abundance and Size of Aquatic Organisms in an Unconstrained Underwater Environment from Continuously Captured Stereo Video

“…However, the authors did not provide detailed information regarding the configuration of the training and evaluation data. In other works, different variants of YOLO were used to detect sablefish Bonofiglio et al [ 32 ], several nordic fish species [ 33 ], reef fishes [ 34 , 35 ] or fish in mangroves [ 36 ].…”