An automatic three dimensional markerless behavioral tracking system of free-moving mice

Han, Yaning; Huang, Kang; Chen, Ke; Wang, Liping; Wei, Pengfei

doi:10.1109/cyber53097.2021.9588299

Cited by 3 publications

(21 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…4e left). The dimensionally reduced distance component by uniform manifold approximation and projection (UMAP) is beneficial to improve the separation of behavior atlas verified by our previous studies [11][12][13][14]32 . But with the increase of data scale, the computational consumption of UMAP would be unacceptable because of limited memory space, which is the second question.…”

Section: Resultsmentioning

confidence: 73%

“…Firstly, the MouseVenue3D automatic camera calibration system is employed to acquire the camera parameters of the camera array (Fig. 2e left top) 11,13 . Then, based on the epipolar constraint of camera parameters, the combination of each animal instance in each camera view is optimized to achieve minimum reprojection error (Fig.…”

Section: Resultsmentioning

confidence: 99%

“…1a). This approach captures the animals covering more view angles and helps to overcome the challenge of frequent occlusion [11][12][13][14] . The camera array is used to capture images of a checkerboard for camera calibration, followed by videos of two free-moving animals for the social behavior test (Video capture phase 1, Fig.…”

Section: Resultsmentioning

confidence: 99%

“…To address this, SBeA employs a multi-view camera array to capture video streams, enabling compensation for the visual field of the cameras and facilitating continuous tracking (Fig. 2b) [11][12][13] . Background and trajectories can be extracted through background subtraction algorithms applied to the raw video streams (Fig.…”

Section: Resultsmentioning

confidence: 99%

“…These data generate millions of labeled frames to train the deep learning-based video instance segmentation (VIS) model. Secondly, to address the issue of occlusion, SBeA employs multiple cameras to capture video streams, which are used to reconstruct 3D poses and prevent complete occlusion [11][12][13] . Thirdly, SBeA resolves the multi-animal identification problem by merging the contour of each animal with the characteristic identity of multiple view angles, which achieves over 90% identification precision without human data annotation.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Social Behavior Atlas: A computational framework for tracking and mapping 3D close interactions of free-moving animals

Han

Chen

Wang

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

The study of social behaviors in animals is essential for understanding their survival and reproductive strategies. However, accurately tracking and analyzing the social interactions of free-moving animals has remained a challenge. Existing multi-animal pose estimation techniques suffer from drawbacks such as the need for extensive manual annotation and difficulty in discriminating between similar-looking animals in close social interactions. In this paper, we present the Social Behavior Atlas (SBeA), a novel computational framework that solves these challenges by employing a deep learning-based video instance segmentation model, 3D pose reconstruction, and unsupervised dynamic behavioral clustering. SBeA framework also involves a multi-camera setup to prevent occlusion, and a novel approach to identify individual animals in close social interactions. We demonstrate the effectiveness of SBeA in tracking and mapping the 3D close interactions of free-moving animals using the example of genetic mutant mice, birds, and dogs. Our results show that SBeA is capable of identifying subtle social interaction abnormalities, and the models and frameworks developed can be applied to a wide range of animal species. SBeA is a powerful tool for researchers in the fields of neuroscience and ecology to study animal social behaviors with a high degree of accuracy and reliability.

show abstract

Section: Resultsmentioning

confidence: 73%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Social Behavior Atlas: A computational framework for tracking and mapping 3D close interactions of free-moving animals

Han

Chen

Wang

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Multi-animal 3D social pose estimation, identification and behaviour embedding with a few-shot learning framework

Han,

Chen,

Wang

et al. 2024

Nat Mach Intell

Self Cite

View full text Add to dashboard Cite

The quantification of animal social behaviour is an essential step to reveal brain functions and psychiatric disorders during interaction phases. While deep learning-based approaches have enabled precise pose estimation, identification and behavioural classification of multi-animals, their application is challenged by the lack of well-annotated datasets. Here we show a computational framework, the Social Behavior Atlas (SBeA) used to overcome the problem caused by the limited datasets. SBeA uses a much smaller number of labelled frames for multi-animal three-dimensional pose estimation, achieves label-free identification recognition and successfully applies unsupervised dynamic learning to social behaviour classification. SBeA is validated to uncover previously overlooked social behaviour phenotypes of autism spectrum disorder knockout mice. Our results also demonstrate that the SBeA can achieve high performance across various species using existing customized datasets. These findings highlight the potential of SBeA for quantifying subtle social behaviours in the fields of neuroscience and ecology.

show abstract

Social Behavior Atlas: A few-shot learning framework for multi-animal 3D social pose estimation, identification, and behavior embedding

Wei

Han

Chen

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

The fact that multi-animal behavior quantification is still technically challenging nowadays greatly limits the accuracy and granularity of social behavior analysis. Data labeling of deep-learning-based approaches can be incredibly laborious, especially when multiple animals closely interact with each other, under which circumstances animal identity switching and body occlusions are likely to happen. To overcome the challenge, we designed a novel framework - Social Behavior Atlas (SBeA) and it shows promising results. SBeA utilizes a much smaller number of labeled frames for multi-animal 3D pose estimation, achieves label-free identification recognition, and successfully applies unsupervised dynamic learning for social behavior classification. Our results also demonstrate that SBeA is capable of achieving high performance across various species using existing customized datasets. Therefore, we can build a multi-animal behavior atlas. To start with, we construct a social behavior atlas for autism spectrum disorder (ASD) knockout mice, and SBeA reveals several behavioral biomarkers which were neglected before. These findings highlight the immense potential of SBeA for quantifying subtle social behaviors and provide a new paradigm in the fields of neuroscience and ecology.

show abstract

An automatic three dimensional markerless behavioral tracking system of free-moving mice

Cited by 3 publications

References 14 publications

Social Behavior Atlas: A computational framework for tracking and mapping 3D close interactions of free-moving animals

Social Behavior Atlas: A computational framework for tracking and mapping 3D close interactions of free-moving animals

Multi-animal 3D social pose estimation, identification and behaviour embedding with a few-shot learning framework

Social Behavior Atlas: A few-shot learning framework for multi-animal 3D social pose estimation, identification, and behavior embedding

Contact Info

Product

Resources

About