Multi-animal pose estimation, identification and tracking with DeepLabCut

Abstract

Estimating the pose of multiple animals is a challenging computer vision problem: frequent interactions cause occlusions and complicate the association of detected keypoints to the correct individuals, as well as having highly similar looking animals that interact more closely than in typical multi-human scenarios. To take up this challenge, we build on DeepLabCut, an open-source pose estimation toolbox, and provide high-performance animal assembly and tracking—features required for multi-animal scenarios. Furthermore, we integrate the ability to predict an animal’s identity to assist tracking (in case of occlusions). We illustrate the power of this framework with four datasets varying in complexity, which we release to serve as a benchmark for future algorithm development.

Using DeepLabCut as a Real-Time and Markerless Tool for Cardiac Physiology Assessment in Zebrafish

“…The deep features allow DLC to extract body parts despite various background challenges or camera distortions [ 47 ]. In addition, DLC possesses a refinement step to take advantage of different scenarios for improving tracking performance [ 48 ]. DLC can predict the points without requiring consistency across the frames.…”

Acoustic camera system for measuring ultrasound communication in mice

“…USV assignment requires frame-by-frame snout locations of each mouse. USVCAM users can choose any available high-precision video tracking software (such as DeepLabCut [ Lauer et al., 2022 ], Social LEAP Estimates Animal Poses [SLEAP; Pereira et al., 2022 ], and Mouse Action Recognition System [MARS; Segalin et al., 2021 ]) to estimate snout locations. In this study, we used AlphaTracker ( Chen et al., 2020 ) for tracking the locations of the snout and the other body parts.…”

Cross-Skeleton Interaction Graph Aggregation Network for Representation Learning of Mouse Social Behaviour

“…1) Intra-skeleton interaction modelling: Similar to [19], [34], we first construct I types of skeleton sequences to learn behavioural information of mice. Following [21], we define the dense physical connections of all the keypoints. Then, we further design a sparse structure where keypoints in the same body part are aggregated into one keypoint by the averaging operation, as shown in Fig.…”

“…1(a). Here, mouse skeleton refers to a list of keypoint connections [21]. Then, a novel Cross-Skeleton Node-level Interaction (CS-NLI) module, shown in Fig.…”