Past, present and future approaches using computer vision for animal re‐identification from camera trap data

Abstract

The ability of a researcher to re‐identify (re‐ID) an individual animal upon re‐encounter is fundamental for addressing a broad range of questions in the study of ecosystem function, community and population dynamics and behavioural ecology. Tagging animals during mark and recapture studies is the most common method for reliable animal re‐ID; however, camera traps are a desirable alternative, requiring less labour, much less intrusion and prolonged and continuous monitoring into an environment. Despite these advantages, the analyses of camera traps and video for re‐ID by humans are criticized for their biases related to human judgement and inconsistencies between analyses. In this review, we describe a brief history of camera traps for re‐ID, present a collection of computer vision feature engineering methodologies previously used for animal re‐ID, provide an introduction to the underlying mechanisms of deep learning relevant to animal re‐ID, highlight the success of deep learning methods for human re‐ID, describe the few ecological studies currently utilizing deep learning for camera trap analyses and our predictions for near future methodologies based on the rapid development of deep learning methods. For decades, ecologists with expertise in computer vision have successfully utilized feature engineering to extract meaningful features from camera trap images to improve the statistical rigor of individual comparisons and remove human bias from their camera trap analyses. Recent years have witnessed the emergence of deep learning systems which have demonstrated the accurate re‐ID of humans based on image and video data with near perfect accuracy. Despite this success, ecologists have yet to utilize these approaches for animal re‐ID. By utilizing novel deep learning methods for object detection and similarity comparisons, ecologists can extract animals from an image/video data and train deep learning classifiers to re‐ID animal individuals beyond the capabilities of a human observer. This methodology will allow ecologists with camera/video trap data to reidentify individuals that exit and re‐enter the camera frame. Our expectation is that this is just the beginning of a major trend that could stand to revolutionize the analysis of camera trap data and, ultimately, our approach to animal ecology.

A computer vision approach for studying fossorial and cryptic crabs

“…For scientists relying on visual observations of organisms or specimens, optical sensors and current advances in machine vision are proving to be useful in opening new possibilities to move ecological research forward (e.g. Matai et al 2012;Dell et al 2014;Konovalov et al 2019;Piechaud et al 2019;Schneider et al 2019). Machine vision and learning has the potential to revolutionize our current sampling methods and analysis in such a way that allow us to address rapidly and efficiently the urgency of sampling natural systems intensively and extensively, hence improving our understanding of these systems and our ability to manage them.…”

“…Computer vision applications have penetrated the field of ecology and proven to be useful in extracting information from images, still or video (e.g. Zion 2012; Spampinato et al 2015;Shafait et al 2016;Villon et al 2018;Weinstein 2018;Schneider et al 2019). There are two main areas in which computer vision is accelerating image analysis for biologists and ecologists.…”

Revisiting animal photo-identification using deep metric learning and network analysis

“…unknown individuals. However, despite the availability of proven and efficient techniques [13] and several successful attempts to apply the method to non-human species [14,15,16,17,18,19,20,21,22,23], re-identification remains a challenging task when applied to animals in wild population where re-observations are limited sensu largo [24].…”

Frame-by-frame annotation of video recordings using deep neural networks

“…Footage captured using video cameras needs to be annotated for use in scientific research, a currently labour intensive process often involving highly trained scientists manually annotating the content of videos frame by frame. Even with dedicated annotation software, this presents a major bottleneck for scientific research based on these data, necessitating the development of computer-assisted approaches (Schneider, Taylor, Linquist, & Kremer, 2019; Weinstein, 2015).…”

“…morphometric features), while spatio-temporal algorithms such as the Cuboid and Harris-3D detectors (Dollár, Rabaud, Cottrell, & Belongie, 2005) capture additional motion information between frames. The main limitations of these approaches arise from their need to know how to represent input features in advance – this requires substantial knowledge of the study species, and hinders generalization across species and environmental contexts (Schneider et al, 2019).…”