SuperAnimal models pretrained for plug-and-play analysis of animal behavior

Ye, Shaokai; Mathis, Alexander; Mathis, Mackenzie Weygandt

doi:10.48550/arxiv.2203.07436

Cited by 12 publications

(12 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first dataset included conventional 2D videos of a single mouse behaving in an open field, with human annotations for four commonly occurring behaviors (locomote, rear, face groom and body groom) (Fig 6a-c). To identify keypoints in this dataset we used DeepLabCut, specifically the TopViewMouse SuperAnimal network from the DLC Model Zoo 31 , which automatically identifies keypoints without the need for annotation data or training. The second dataset (part of the CalMS21 benchmark 30 ) included a set of three manually annotated social behaviors (mounting, investigation, and attack) as well as keypoints for a pair of interacting mice (Fig 6d-f).…”

Section: Resultsmentioning

confidence: 99%

Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Weinreb

Osman

Zhang

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Keypoint tracking algorithms have revolutionized the analysis of animal behavior, enabling investigators to flexibly quantify behavioral dynamics from conventional video recordings obtained in a wide variety of settings. However, it remains unclear how to parse continuous keypoint data into the modules out of which behavior is organized. This challenge is particularly acute because keypoint data is susceptible to high frequency jitter that clustering algorithms can mistake for transitions between behavioral modules. Here we present keypoint-MoSeq, a machine learning-based platform for identifying behavioral modules ("syllables") from keypoint data without human supervision. Keypoint-MoSeq uses a generative model to distinguish keypoint noise from behavior, enabling it to effectively identify syllables whose boundaries correspond to natural sub-second discontinuities inherent to mouse behavior. Keypoint-MoSeq outperforms commonly-used alternative clustering methods at identifying these transitions, at capturing correlations between neural activity and behavior, and at classifying either solitary or social behaviors in accordance with human annotations. Keypoint-MoSeq therefore renders behavioral syllables and grammar accessible to the many researchers who use standard video to capture animal behavior.

show abstract

Section: Resultsmentioning

confidence: 99%

Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Weinreb

Osman

Zhang

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Semi-supervised learning is not the only technique that enables improvements over standard supervised learning protocols. First, it has been suggested that supervised pose estimation networks can be improved by pretraining them on large labeled datasets for image classification [9] or pose estimation [47], to an extent that might eliminate dataset-specific training [48]. Other work avoids pretraining altogether by using lighter architectures [10].…”

Section: Discussionmentioning

confidence: 99%

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools

Biderman

Whiteway

Hurwitz

et al. 2023

Preprint

View full text Add to dashboard Cite

Pose estimation algorithms are shedding new light on animal behavior and intelligence. Most existing models are only trained with labeled frames (supervised learning). Although effective in many cases, the fully supervised approach requires extensive image labeling, struggles to generalize to new videos, and produces noisy outputs that hinder downstream analyses. We address each of these limitations with a semi-supervised approach that leverages the spatiotemporal statistics of unlabeled videos in two different ways. First, we introduce unsupervised training objectives that penalize the network whenever its predictions violate smoothness of physical motion, multiple-view geometry, or depart from a low-dimensional subspace of plausible body configurations. Second, we design a new network architecture that predicts pose for a given frame using temporal context from surrounding unlabeled frames. These context frames help resolve brief occlusions or ambiguities between nearby and similar-looking body parts. The resulting pose estimation networks achieve better performance with fewer labels, generalize better to unseen videos, and provide smoother and more reliable pose trajectories for downstream analysis; for example, these improved pose trajectories exhibit stronger correlations with neural activity. We also propose a Bayesian post-processing approach based on deep ensembling and Kalman smoothing that further improves tracking accuracy and robustness. We release a deep learning package that adheres to industry best practices, supporting easy model development and accelerated training and prediction. Our package is accompanied by a cloud application that allows users to annotate data, train networks, and predict new videos at scale, directly from the browser.

show abstract

“…In our final evaluation, we sought to benchmark DAMM against an existing method for mouse localization, the SuperAnimal-TopViewMouse model released by DeepLabCut (SA-DLC) (16). This tool predicts body keypoint trajectories and is aimed at generalization.…”

Section: Cmentioning

confidence: 99%

“…For the comparison to the SuperAnimal-TopViewMouse model released by DeepLabCut (16), we used predictions aggregated over scales [200,300,400,500,600] which was the only hyperparameter selected by the end-user. To construct a bounding box that is used to approximate a bounding box localization, we compute the tightest box encompassing all points, while excluding all tail points.…”

Section: Model Selection and Trainingmentioning

confidence: 99%

DAMM for the detection and tracking of multiple animals within complex social and environmental settings

Kaul,

McDevitt,

Johnson

et al. 2024

Preprint

View full text Add to dashboard Cite

Accurate detection and tracking of animals across diverse environments are crucial for behavioral studies in various disciplines, including neuroscience. Recently, machine learning and computer vision techniques have become integral to the neuroscientist's toolkit, enabling high-throughput behavioral studies. Despite advancements in localizing individual animals in simple environments, the task remains challenging in complex conditions due to intra-class visual variability and environmental diversity. These limitations hinder studies in ethologically-relevant conditions, such as when animals are concealed within nests or in obscured environments. Moreover, current tools are laborious and time-consuming to employ, requiring extensive, setup-specific annotation and model training/validation procedures. To address these challenges, we introduce the 'Detect Any Mouse Model' (DAMM), a pretrained object detector for localizing mice in complex environments, capable of robust performance with zero to minimal additional training on new experimental setups. Our approach involves collecting and annotating a diverse dataset that encompasses single and multi-housed mice in various lighting conditions, experimental setups, and occlusion levels. We utilize the Mask R-CNN architecture for instance segmentation and validate DAMM's performance with no additional training data (zero-shot inference) and with few examples for fine-tuning (few-shot inference). DAMM excels in zero-shot inference, detecting mice, and even rats, in entirely unseen scenarios and further improves with minimal additional training. By integrating DAMM with the SORT algorithm, we demonstrate robust tracking, competitively performing with keypoint-estimation-based methods. Finally, to advance and simplify behavioral studies, we made DAMM accessible to the scientific community with a user-friendly Python API, shared model weights, and a Google Colab implementation.

show abstract

SuperAnimal models pretrained for plug-and-play analysis of animal behavior

Cited by 12 publications

References 49 publications

Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools

DAMM for the detection and tracking of multiple animals within complex social and environmental settings

Contact Info

Product

Resources

About