Automatic discard registration in cluttered environments using deep learning and object tracking: class imbalance, occlusion, and a comparison to human review

Abstract

This paper presents and evaluates a method for detecting and counting demersal fish species in complex, cluttered, and occluded environments that can be installed on the conveyor belts of fishing vessels. Fishes on the conveyor belt were recorded using a colour camera and were detected using a deep neural network. To improve the detection, synthetic data were generated for rare fish species. The fishes were tracked over the consecutive images using a multi-object tracking algorithm, and based on multiple observations, the fish species was determined. The effect of the synthetic data, the amount of occlusion, and the observed dorsal or ventral fish side were investigated and a comparison with human electronic monitoring (EM) review was made. Using the presented method, a weighted counting error of 20% was achieved, compared to a counting error of 7% for human EM review on the same recordings.

Length estimation of fish detected as non-occluded using a smartphone application and deep learning techniques

“…Second, there are difficulties in identifying fish species using deep learning techniques. Identification of Fish species using deep learning techniques involves families and genera (e.g., van Essen et al, 2021). Some examples have been reported as possible situations to identify fish species, although the study is based on a few (ten classes) fish species (Lu et al, 2020), or the surrounding environment of imaging is fixed on the ship (Ovalle et al, 2022).…”

Electronic monitoring in fisheries

“…In case, implementation of AI will be accomplished, cost will be reduced, which, eventually, makes EM accessible for a large share of the world's fisheries . Review of EM studies worldwide indicated that developments on automated catch registration through computer vision technology are picking up (van Helmond et al, 2021). So far, preliminary results of projects to integrate computer vision technology are promising, also when the conditions to register catch per species are more challenging, such as detecting and counting demersal fish species in complex, cluttered, and occluded environments that can be installed on the conveyor belts of fishing vessels (van Essen et al, 2021;ICES, 2022).…”

“…This can be fuelled by indirect incentives, such as increased market access through eco-labelling and certification, but also by experiencing advantages in terms of better fishing opportunities, e.g. data sharing and increased insight in fishing activity, increased transparency, real-time fisheries management, result-based management Michelin and Zimring, 2020;van Helmond et al, 2021;Steins et al, 2022). These circumstances can only be realised with implementation of AI to establish a considerable reduction in running costs of EM and rapid data analysis.…”

“…First trials indicate the potential of this technology to automate the process of counting fish by species and size without interference of the sorting procedures on board. The algorithms, which form the basis for this technology, showed the ability to recognize and record fish correctly during the sorting process on the conveyer belt, also is situations where fish overlap or are partly covered with debris and benthic organic material (French et al, 2019;van Essen et al, 2021). Also, the ongoing innovation about dirt detection on camera lenses is a promising development in being less dependent of onboard crew to maintain the cameras contributes to the development of more self-sufficient and robust EM systems.…”

“…The best possible way to reduce costs is automatization of the manual review process. Computer vision technology to automatically and immediately analyse the collected video data of catches onboard fishing vessels is developing (French et al, 2019;van Essen et al, 2021). The expectation is that computer vison technology, more broadly described as artificial intelligence (AI), will be capable of preforming species identification and volume estimation onboard vessels, cost-effectively delivering "real-time" data on catch and fishing activity (chapter 5; Michelin and Zimring, 2020).…”