Validation methods for plankton image classification systems

Abstract

In recent decades, the automatic study and analysis of plankton communities using imaging techniques has advanced significantly. The effectiveness of these automated systems appears to have improved, reaching acceptable levels of accuracy. However, plankton ecologists often find that classification systems do not work as well as expected when applied to new samples. This paper proposes a methodology to assess the efficacy of learned models which takes into account the fact that the data distribution (the plankton composition of the sample) can vary between the model building phase and the production phase. As opposed to most validation methods that consider the individual organism as the unit of validation, our approach uses a validation‐by‐sample, which is more appropriate when the objective is to estimate the abundance of different morphological groups. We argue that, in these cases, the base unit to correctly estimate the error is the sample, not the individual. Thus, model assessment processes require groups of samples with sufficient variability in order to provide precise error estimates.

Improving plankton image classification using context metadata

“…Prior to the development of CNNs, plankton images were classified with varying degrees of success primarily using geometric features (reviewed in González et al ). Recently, CNNs have been applied to plankton classification problems hinting at the potential of the approach (Wang et al ; Zheng et al ).…”

“…Supervised Machine Learning algorithms depend on training data being representative of future samples. For plankton image classification, this guidance is applicable not only for the distribution of the sampled organisms (González et al ), but also for any context metadata used. The term “concept drift” (Widmer and Kubat ; González et al ) describes the condition when this future distribution is not stationary.…”

“…For plankton image classification, this guidance is applicable not only for the distribution of the sampled organisms (González et al ), but also for any context metadata used. The term “concept drift” (Widmer and Kubat ; González et al ) describes the condition when this future distribution is not stationary. Some of the metadata distributions will drift faster than the images of the individuals themselves, as the population level responses can lag the changes in context measurements.…”

“…It is important that the images selected for annotation are selected without bias. This topic and other best practices for validating feature‐based classification of plankton images are discussed by González et al (). Our images were drawn from seasonal Zooglider coverage consisting of 225 dive profiles containing 1.45 million full frame image captures.…”

Automatic plankton quantification using deep features

“…We are interested in estimating the prevalence of each class in an unknown water sample. To that end, we have used quantification algorithms with deep features as their input, and we have analyzed their performance with a rigorously designed validation methodology (González et al, 2017a) where the sample is the minimum test unit.…”