Anti-drift pose tracker (ADPT): A transformer-based network for robust animal pose estimation cross-species

Abstract

Deep learning-based methods for animal pose estimation have recently made substantial progress in improving the accuracy and efficiency of quantitative descriptions of animal behavior. However, these methods commonly suffer from tracking drifts, i.e., sudden jumps in the estimated position of a body point due to noise, thus reducing the reliability of behavioral study results. Here, we present a transformer-based animal pose estimation tool, called Anti-Drift Pose Tracker (ADPT), for eliminating tracking drifts in behavior analysis. To verify the anti-drift performance of ADPT, we conduct extensive experiments in multiple cross-species datasets, including long-term recorded mouse and monkey behavioral datasets collected by ourselves, as well as two public Drosophilas and macaques datasets. Our results show that ADPT greatly reduces the rate of tracking drifts, and significantly outperforms the existing deep-learning methods, such as DeepLabCut, SLEAP, and DeepPoseKit. Moreover, ADPT is compatible with multi-animal pose estimation, enabling animal identity recognition and social behavioral study. Specifically, ADPT provided an identification accuracy of 93.16% for 10 unmarked mice, and of 90.36% for free-social unmarked mice which can be further refined to 99.72%. Compared to other multi-stage network-based tools like multi-animal DeepLabCut, SIPEC and Social Behavior Atlas, the end-to-end structure of ADPT supports its lower computational costs and meets the needs of real-time analysis. Together, ADPT is a versatile anti-drift animal behavior analysis tool, which can greatly promote the accuracy, robustness, and reproducibility of animal behavioral studies. The code of ADPT is available at https://github.com/tangguoling/ADPT.