Measuring agreement among experts in classifying camera images of similar species

Abstract

Camera trapping and solicitation of wildlife images through citizen science have become common tools in ecological research. Such studies collect many wildlife images for which correct species classification is crucial; even low misclassification rates can result in erroneous estimation of the geographic range or habitat use of a species, potentially hindering conservation or management efforts. However, some species are difficult to tell apart, making species classification challenging—but the literature on classification agreement rates among experts remains sparse. Here, we measure agreement among experts in distinguishing between images of two similar congeneric species, bobcats (Lynx rufus) and Canada lynx (Lynx canadensis). We asked experts to classify the species in selected images to test whether the season, background habitat, time of day, and the visible features of each animal (e.g., face, legs, tail) affected agreement among experts about the species in each image. Overall, experts had moderate agreement (Fleiss’ kappa = 0.64), but experts had varying levels of agreement depending on these image characteristics. Most images (71%) had ≥1 expert classification of “unknown,” and many images (39%) had some experts classify the image as “bobcat” while others classified it as “lynx.” Further, experts were inconsistent even with themselves, changing their classifications of numerous images when they were asked to reclassify the same images months later. These results suggest that classification of images by a single expert is unreliable for similar‐looking species. Most of the images did obtain a clear majority classification from the experts, although we emphasize that even majority classifications may be incorrect. We recommend that researchers using wildlife images consult multiple species experts to increase confidence in their image classifications of similar sympatric species. Still, when the presence of a species with similar sympatrics must be conclusive, physical or genetic evidence should be required.

Will Lynx Lose Their Edge? Canada Lynx Occupancy in Washington

“…Trained personnel identified animals captured in photographs to species, and an expert reviewer verified species (T. W. King or D. Thornton served as the expert). Although the consistency of classifying images of lynx and the similar‐looking bobcat (which also inhabits the study area) from single photographs has been called into question (Gooliaff and Hodges 2018), we found high levels of agreement in identification of lynx from the image bursts used in our study, which provided multiple views of easily distinguishable features such as a short fully black tail tip and large paws (Thornton et al 2019). Thus, the potential for mis‐classification of lynx in this study is low.…”

Just-in-Time Training Improves Accuracy of Citizen Scientist Wildlife Identifications from Camera Trap Photos

“…For each identification for each participant, we calculated accuracy by comparing the participant's identification to our official identification. To increase confidence in the accuracy of the official identification, identification was determined, using photos in bursts of 5 to verify the observation, and corroborated by each of the three authors prior to image upload onto Zooniverse (Gooliaff and Hodges 2018). When calculating accuracy of participants, "Don't Know" and "Other" responses were categorized as incorrect.…”

Reassessing the success of experts and nonexperts at correctly differentiating between closely related species from camera trap images: A reply to Gooliaff and Hodges

“…In a recent issue of Ecology and Evolution , Gooliaff and Hodges () presented the results of a study that measured agreement in classifying images of similar species from photographic records (“Measuring agreement among experts in classifying camera images of similar species,” Ecology and Evolution https://doi.org/10.1002/ece3.4567). Gooliaff and Hodges used an example of classifying images of Canada lynx ( Lynx canadensis ) and bobcat ( Lynx rufus ), which are similar mid‐sized felids that are sympatric in several areas of the United States and Canada.…”