DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning

Abstract

Quantitative behavioral measurements are important for answering questions across scientific 13 disciplines-from neuroscience to ecology. State-of-the-art deep-learning methods offer major advances in 14 data quality and detail by allowing researchers to automatically estimate locations of an animal's body parts 15 directly from images or videos. However, currently-available animal pose estimation methods have limitations 16 in speed and robustness. Here we introduce a new easy-to-use software toolkit, DeepPoseKit, that addresses 17 these problems using an efficient multi-scale deep-learning model, called Stacked DenseNet, and a fast 18 GPU-based peak-detection algorithm for estimating keypoint locations with subpixel precision. These advances 19 improve processing speed >2× with no loss in accuracy compared to currently-available methods. We 20 demonstrate the versatility of our methods with multiple challenging animal pose estimation tasks in laboratory 21 and field settings-including groups of interacting individuals. Our work reduces barriers to using advanced 22 tools for measuring behavior and has broad applicability across the behavioral sciences. 23 24 42 et al. (2018) and Pereira et al. (2019), who make use of a popular type of machine learning model called 43 convolutional neural networks, or CNNs (LeCun et al. 2015; Appendix 1), to automatically measure detailed 44 representations of animal posture-structural keypoints, or joints, on the animal's body-directly from images 45 1 of 36and without markers. While these methods offer a major advance over conventional methods with regard 46 to data quality and detail, they have disadvantages in terms of speed and robustness, which may limit their 47 157 inference speed for the DeepLabCut model (Mathis et al., 2018) can be improved by decreasing the resolution 158 4 of 36 235 models to the models from Mathis et al. (2018) (DeepLabCut) and Pereira et al. (2019) (LEAP) in terms of speed, 236 accuracy, training time, and generalization ability. We benchmarked these models using three image datasets 237 recorded in the laboratory and the field-including multiple interacting individuals that were first localized and 238 cropped from larger, multi-individual images (see "Methods" for details). 239 6 of 36 654 and TensorFlow teams, and Alexander Jung for their open source contributions, which provided the core 655programming interface for our work. We thank A. Strandburg-Peshkin, Vivek H. Sridhar, Michael L. Smith, and 656 Joseph B. Bak-Coleman for their helpful discussions on the project and comments on the manuscript. We 657 also thank M.L.S. for the use of his GPU. We thank Felicitas Oehler for annotating the zebra posture data and 658 Chiara Hirschkorn for assistance with filming the locusts and annotating the locust posture data. We thank 659

Simple Behavioral Analysis (SimBA) – an open source toolkit for computer classification of complex social behaviors in experimental animals

“…There are, however, many other effective open-source solutions for animal tracking and pose-estimation 20,22,69 and SimBA is agnostic to the tools used to extrapolate positional coordinates. SimBA also has an interface for DeepPoseKit 26 that permits a range of alternative neural network architectures for pose-estimation that may offer speed and accuracy advantages 70 . Others have also successfully implemented YOLO-based approaches 71 .…”

“…Such methods can permit online classifications in semi-natural environments and are a foundation for impending closed-loop monitoring and forecasting systems 25 in behavioural neuroscience, but do require significant investment in specialized hardware and a working knowledge of computer science approaches. Parallel advances in animal tracking have produced accessible open-source pose-estimation tools for accurate tracking of experimenter-defined body-parts in noisy and variable environments [26][27][28][29] (Table 1). These open-source initiatives have proven to be both far less expensive to execute, and provide better animal tracking outcomes, than currently available commercial products 30 .…”

A machine-vision approach for automated pain measurement at millisecond timescales

“…In kind, many researchers are taking steps backwards to first properly understand the component parts of a complex behavioral sequence before proceeding to identify the neuronal correlates that drive those motor patterns. Both supervised and unsupervised machine learning algorithms are now able to follow unlabeled individual limbs on an experimental animal and automatically define behaviors of interest [12,14,15,[24][25][26]. Although the majority of these tools have yet to be adopted in mass by the pain research community, some of this technology is already in use by pain researchers.…”

Markerless tracking of an entire insect colony

“…Recent machine vision advances in the precise posture tracking of individual animals 17,18,52 as well as of the positions of highly-similar organisms in groups 46 are enabling new quantitative studies of behavior 53,54 . In collective behavior specifically, the use of CNNs for the pixel-based identification of individual organisms has significantly advanced markerless, long-time tracking in 2D, from more modest assemblies (~10 individuals) 34 to large groups (~100 individuals) 46 .…”