Automatic fish species classification in underwater videos: exploiting pre-trained deep neural network models to compensate for limited labelled data

Siddiqui, Shoaib Ahmed; Ahmad, Sahrim; Malik, Muhammad Imran; Shafait, Faisal; Mian, Ajmal; Shortis, Mark R.; Harvey, Euan S.

doi:10.1093/icesjms/fsx109

Cited by 179 publications

(112 citation statements)

References 31 publications

Supporting

Mentioning

111

Contrasting

Unclassified

Order By: Relevance

“…There are often commercial restrictions on access to footage and publication of the data obtained from these sources that will need to be reconciled. Nonetheless, we anticipate that scientific inference from natural history video could increase in future as recording equipment able to capture high-resolution imagery becomes more accessible 6 and advances in computer vision and machine-learning enables automated species recognition 41,42 .…”

Section: Resultsmentioning

confidence: 99%

Natural history footage provides new reef fish biodiversity information for a pristine but rarely visited archipelago

Liggins

Sweatman

Trnski³

et al. 2020

Sci Rep

View full text Add to dashboard Cite

There remain parts of our planet that are seldom visited by humans, let alone scientists. In such locations, crowd-sourced or citizen scientist data can be critical in describing biodiversity and detecting change. Rangitāhua, the Kermadec Islands, are 750 km from the nearest human-habitation. Although our knowledge of this near pristine location has increased with recent biodiversity expeditions, we still lack comprehensive understanding of the marine biodiversity surrounding the islands. In 2015, professional underwater videographers were commissioned to produce a nature documentary focused on Rangitāhua's reefs. We strategically surveyed the raw documentary video and examined how biodiversity estimates differed from traditional scientific surveys. We uncovered three new fish species records for Rangitāhua, extending the known distribution for each species, two of which are also new records for New Zealand waters. Comparison of documentary video footage with scientific survey methods showed that estimates of reef fish species richness from the documentary video were similar to stationary surveys, but lower than non-stationary surveys. Moreover, all survey methods, including documentary video, captured different fish assemblages, reflecting each method's particular bias. Overall, we provide a proof-of-concept for how collaborations between scientists and professional natural historians, such as videographers and photographers, can provide valuable biodiversity information.Despite the efforts of scientists to catalogue the global distribution of biodiversity, we continue to record species in new locations (the "Wallacean shortfall" 1 ; reviewed in 2 ). New species records change our understanding of individual species ranges and improve our biodiversity knowledge for a given location. Our knowledge of species distributions in the ocean is particularly incomplete 3,4 . Even for many relatively well-researched coastal environments, new species records continue to accumulate. In remote locations, biodiversity inventories are often inadequate due to the logistical and financial challenges of implementing long-term research or monitoring 5 . Biodiversity surveys in such areas are often opportunistic and lack continuity, meaning that scientists and managers wishing to understand the biodiversity values of remote locations have to be resourceful when sourcing data.Alternative means of biodiversity cataloging are now commonplace as modern biodiversity informatics platforms (such as: iNaturalist, www.inaturalist.org; the Range Extension Database and Mapping project, REDMAP, www.redmap.org.au) are geared toward harnessing the collective knowledge of natural history observers in our society. In order for this citizen-sourced data to be incorporated into biodiversity research, there must be some form of quality control 6 . Verification of citizen-generated biodiversity records typically involves a professional scientist checking photographs and associated sighting metadata for accuracy. The citizen-sourced data may then be co...

show abstract

Section: Resultsmentioning

confidence: 99%

Natural history footage provides new reef fish biodiversity information for a pristine but rarely visited archipelago

Liggins

Sweatman

Trnski³

et al. 2020

Sci Rep

View full text Add to dashboard Cite

show abstract

“…Note the significant increase in image training complexity compared to the 32 × 32 × 1 gray images of [1], [13]. The following three CNN architectures were used in [2]: AlexNet [16], VGGNet [27], and ResNet [28]. The CNNs' original layers were initialized by loading the weights pre-trained on the ImageNet's [29] vast collection of images, which is commonly referred to as the transfer learning or knowledge transfer setup or technique [30], [31].…”

Section: A Related Workmentioning

confidence: 99%

“…The CNNs' original layers were initialized by loading the weights pre-trained on the ImageNet's [29] vast collection of images, which is commonly referred to as the transfer learning or knowledge transfer setup or technique [30], [31]. An ImageNet-trained CNN often exhibits superior performance compared to the same but randomly initialized CNN, when the CNN is retrained and/or re-purposed for different classes of images [31]; for example, the 16 fish species in [2]. The three considered ImageNet-trained CNNs were applied without further training to extract image features and then [2] used the features as input into a standard SVM classifier.…”

Section: A Related Workmentioning

confidence: 99%

“…An ImageNet-trained CNN often exhibits superior performance compared to the same but randomly initialized CNN, when the CNN is retrained and/or re-purposed for different classes of images [31]; for example, the 16 fish species in [2]. The three considered ImageNet-trained CNNs were applied without further training to extract image features and then [2] used the features as input into a standard SVM classifier. Out of the considered three CNNs, the ImageNet pre-trained ResNet [28] together with the SVM classifier achieved the best accuracy of 89% on the testing subset of 663 images.…”

Section: A Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Underwater Fish Detection with Weak Multi-Domain Supervision

Konovalov

Saleh

Bradley

et al. 2019

2019 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Given a sufficiently large training dataset, it is relatively easy to train a modern convolution neural network (CNN) as a required image classifier. However, for the task of fish classification and/or fish detection, if a CNN was trained to detect or classify particular fish species in particular background habitats, the same CNN exhibits much lower accuracy when applied to new/unseen fish species and/or fish habitats. Therefore, in practice, the CNN needs to be continuously fine-tuned to improve its classification accuracy to handle new project-specific fish species or habitats. In this work we present a labellingefficient method of training a CNN-based fish-detector (the Xception CNN was used as the base) on relatively small numbers (4,000) of project-domain underwater fish/no-fish images from 20 different habitats. Additionally, 17,000 of known negative (that is, missing fish) general-domain (VOC2012) above-water images were used. Two publicly available fish-domain datasets supplied additional 27,000 of above-water and underwater positive/fish images. By using this multi-domain collection of images, the trained Xception-based binary (fish/not-fish) classifier achieved 0.17% false-positives and 0.61% false-negatives on the project's 20,000 negative and 16,000 positive holdout test images, respectively. The area under the ROC curve (AUC) was 99.94%.

show abstract

“…Traditional fishing activities, the rapid development of underwater renewable energy plants, the expansion of fish farms and the increasing requirements for surveying ageing infrastructures (e.g., dams, water reservoirs, harbour facilities); all call for rapid technological advances in underwater observation technologies. For example, in fisheries and aquaculture, there is a need of high quality and real-time data on, namely, fish abundances and size, as well as identification of marine habitat status, to achieve the sustainable management of these resources [25][26][27]. Food supply from the ocean now contribute more than 15% of the overall amount of animal protein consumed worldwide [28], hence it is essential for human health and wellbeing that these resources are managed at sustainable levels.…”

Section: Surveying the Oceanmentioning

confidence: 99%

Range-Gated Imaging System for Underwater Monitoring in Ocean Environment

et al. 2018

View full text Add to dashboard Cite

High-quality video observations are very much needed in underwater environments for the monitoring of several ecosystem indicators and to support the sustainable development and management of almost all activities in the ocean. Reliable video observations are however challenging to collect, because of the generally poor visibility conditions and the difficulties to deploy cost-effective sensors and platforms in the marine environment. Visibility in water is regulated by natural light availability at different depths, and by the presence of suspended particles, scattering incident light in all directions. Those elements are also largely variable in time and space, making it difficult to identify technological solutions that can be used in all conditions. By combining state-of-the-art "time of flight" (ToF) image sensors and innovative pulsed laser illumination, we have developed a range-gated camera system (UTOFIA) that enables affordable and enhanced 3D underwater imaging at high resolution. This range-gated solution allows users to eliminate close-range backscattering, improving quality of the images and providing information on the distance of each illuminated object, hence giving access to real-time 3D measurements. Furthermore, as the system is based on pulsed laser light, it is almost independent of natural light conditions and can achieve similar performances at an extended depth range. We use this system to collect observations in different oceanographic conditions and for different applications, including aquaculture monitoring, seafloor mapping, litter identifications and structure inspection. Performances are evaluated by comparing images to regular cameras and by using standard targets to assess accuracy and precision of distance measurements. We suggest that this type of technology can become a standard in underwater 3D imaging to support the future development of the ocean economy.Sustainability 2019, 11, 162 2 of 13 create a turbid environment that strongly increases light scattering and enhances the absorption probability of photons [3]. When the light source is the sun, this process effectively decreases the amount of ambient light present at any depth and limits the range of visual observations. With artificial illumination, the range for underwater vision can be extended (for example we can move deeper) but at the cost of degrading image contrast due to (forward-and back-) scattering generated by suspended particles. The situation is similar to driving a car in foggy conditions with the headlights on: increasing the power of illumination does not improve the visibility, as the backscattering increases proportionally. Image contrast is also lowered with shorter visual range as the light attenuation will reduce the illumination of distant targets. These factors remain the outstanding challenges in underwater imaging and limit the application of visual observations in many sectors [1,2,4].Various optical and acoustic imaging systems for mitigating or solving these problems are under constant develo...

show abstract

Automatic fish species classification in underwater videos: exploiting pre-trained deep neural network models to compensate for limited labelled data

Cited by 179 publications

References 31 publications

Natural history footage provides new reef fish biodiversity information for a pristine but rarely visited archipelago

Natural history footage provides new reef fish biodiversity information for a pristine but rarely visited archipelago

Underwater Fish Detection with Weak Multi-Domain Supervision

Range-Gated Imaging System for Underwater Monitoring in Ocean Environment

Contact Info

Product

Resources

About