Accuracy and consistency of grass pollen identification by human analysts using electron micrographs of surface ornamentation

Abstract

• Premise of the study: Humans frequently identify pollen grains at a taxonomic rank above species. Grass pollen is a classic case of this situation, which has led to the development of computational methods for identifying grass pollen species. This paper aims to provide context for these computational methods by quantifying the accuracy and consistency of human identification.• Methods: We measured the ability of nine human analysts to identify 12 species of grass pollen using scanning electron microscopy images. These are the same images that were used in computational identifications. We have measured the coverage, accuracy, and consistency of each analyst, and investigated their ability to recognize duplicate images.• Results: Coverage ranged from 87.5% to 100%. Mean identification accuracy ranged from 46.67% to 87.5%. The identification consistency of each analyst ranged from 32.5% to 87.5%, and each of the nine analysts produced considerably different identification schemes. The proportion of duplicate image pairs that were missed ranged from 6.25% to 58.33%.• Discussion: The identification errors made by each analyst, which result in a decline in accuracy and consistency, are likely related to psychological factors such as the limited capacity of human memory, fatigue and boredom, recency effects, and positivity bias.

Poaceae Pollen from Southern Brazil: Distinguishing Grasslands (Campos) from Forests by Analyzing a Diverse Range of Poaceae Species

“…Some of these species also inhabit southern Brazil. Some studies (Table 1) have explored the surface of the Poaceae pollen grains through SEM, which has contributed to separation of taxonomic groups (Köhler and Lange, 1979; Linder and Ferguson, 1985; Chaturvedi et al, 1994, 1998; Chaturvedi and Datta, 2001; Skvarla et al, 2003; Datta and Chaturvedi, 2004; Liu et al, 2004, 2005; Perveen, 2006; Kashikar and Kalkar, 2010; Ahmad et al, 2011; Dórea, 2011; Perveen and Qaiser, 2012; Mander et al, 2013, 2014; Nazir et al, 2013; Morgado et al, 2015; Needham et al, 2015; Mander and Punyasena, 2016). However, other studies using light microscopy (LM) have shown morphometric differences in the size of the Poaceae pollen grain species (Salgado-Labouriau and Rinaldi, 1990; Katsiotis and Forsberg, 1995; Joly et al, 2007; Schüler and Behling, 2011a,b; Jan et al, 2014).…”

On the Taxonomic Resolution of Pollen and Spore Records of Earth’s Vegetation

“…For example, seven human subjects classified 120 SEM images of 12 grass pollen species with 68%-82% accuracy (10 images per species), but the classification schemes of each subject were very different from one another, and consistency between subjects was just 28% (Mander et al 2013). A more recent analysis of human performance on this data set also highlighted considerable differences between analysts and notable self-inconsistency as well (Mander et al 2014). Similar investigations of human classification consistency with a marine microfossil group revealed irreconcilable differences between analysts (Ginsburg 1997).…”

“…1E). The SEM has been used extensively in palynology because of its ability to image structures that are just a few nanometers in size, and although specimen preparation and image acquisition time have been barriers to the use of the SEM in routine palynological studies (e.g., Faegri et al 1992), recent work has shown that SEM images of surface ornamentation can be used by human analysts to classify species of grass pollen (Mander et al 2013(Mander et al , 2014. Von Post's comment also foreshadows the idea that different microscopy techniques are suited to the analysis of different morphological features of sporomorphs (Sivaguru et al 2012).…”

“…Typically, palynologists examine by eye the sizes and shapes of attributes that are thought to have taxonomic significance, such as the sculptural elements on the sporomorph surface, and then use these attributes to qualitatively compare unknown specimens to classified reference material. Experiments designed to measure the consistency of classifications undertaken in this way raise doubts about the ability of human analysts to consistently repeat such classifications over the course of an entire study that may encompass tens of thousands of specimens (e.g., Culverhouse 2007;MacLeod et al 2010;Mander et al 2014). For example, seven human subjects classified 120 SEM images of 12 grass pollen species with 68%-82% accuracy (10 images per species), but the classification schemes of each subject were very different from one another, and consistency between subjects was just 28% (Mander et al 2013).…”

“…We suspect that expert palynologists would fare well in such a test and, by extension, that they generate consistent classifications at the genus or family level in Quaternary time. However, as testing suggests (Mander et al 2013(Mander et al , 2014, we have little doubt that palynologists struggle to generate species-level classifications of problematic groups that are both self-consistent and consistent between workers, and this view has led to calls for an algorithmic approach to classification that employs computational image analyses and machine learning to quantify small differences between morphologically similar species (MacLeod 2010;MacLeod et al 2010;Punyasena et al 2012;Mander et al 2013). …”