SLEAP: A deep learning system for multi-animal pose tracking

Abstract

The desire to understand how the brain generates and patterns behavior has driven rapid methodological innovation in tools to quantify natural animal behavior. While advances in deep learning and computer vision have enabled markerless pose estimation in individual animals, extending these to multiple animals presents unique challenges for studies of social behaviors or animals in their natural environments. Here we present Social LEAP Estimates Animal Poses (SLEAP), a machine learning system for multi-animal pose tracking. This system enables versatile workflows for data labeling, model training and inference on previously unseen data. SLEAP features an accessible graphical user interface, a standardized data model, a reproducible configuration system, over 30 model architectures, two approaches to part grouping and two approaches to identity tracking. We applied SLEAP to seven datasets across flies, bees, mice and gerbils to systematically evaluate each approach and architecture, and we compare it with other existing approaches. SLEAP achieves greater accuracy and speeds of more than 800 frames per second, with latencies of less than 3.5 ms at full 1,024 × 1,024 image resolution. This makes SLEAP usable for real-time applications, which we demonstrate by controlling the behavior of one animal on the basis of the tracking and detection of social interactions with another animal.

Selfee, self-supervised features extraction of animal behaviors

“…Then, we would like to know if our neural network could capture fine-grained features beyond human-defined labels. Using cutting-edge animal tracking software SLEAP ( Pereira et al, 2022 ), flies’ wings, heads, tails, and thoraxes were tracked throughout the clip automatically with manual proofreading ( Figure 3—video 1 ). Three straight-forward features were visualized on the t-SNE map.…”

Cerebellar contributions to a brainwide network for reversal learning

“…Our pose analysis (Klibaite et al, 2022) enabled us to simultaneously analyze the detailed kinematics of limb movement and longer time-scale features of behavior. Such an analysis required a method that could track individual body parts, such as LEAP (Pereira et al, 2022, 2019) or other approaches (Wiltschko et al, 2020, 2015); (Mathis et al, 2018; Nath et al, 2019). We did not find differences in gait or in spatial occupancy of the arena, suggesting that the chosen cerebellar perturbations affected the evolution of motor behaviors over several days, but not the capacity to interact with the physical environment or generate locomotor behavior.…”

Floral humidity as a signal – not a cue – in a nocturnal pollination system

“…We used an animal-pose tracking software SLEAP 82,83 to track pre-selected body parts on the moth to collect information on moth position and behavioral choice in the two-choice behavioral assays. We selected the eye, head, proboscis tip, thorax, wingtips, and the abdominal tip of the moth’s body for tracking-based analyses on videos obtained by the motion-sensing camera (Fig.…”