MouseVenue3D: A Markerless Three-Dimension Behavioral Tracking System for Matching Two-Photon Brain Imaging in Free-Moving Mice

Han, Yaning; Huang, Kang; Chen, Ke; Pan, Hongli; Ju, Furong; Long, Yueyue; Gao, Gao; Wu, Runlong; Wang, Aimin; Wang, Liping; Wei, Pengfei

doi:10.1007/s12264-021-00778-6

Cited by 12 publications

(32 citation statements)

References 81 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…4e left). The dimensionally reduced distance component by uniform manifold approximation and projection (UMAP) is beneficial to improve the separation of behavior atlas verified by our previous studies 11–14,32 . But with the increase of data scale, the computational consumption of UMAP would be unacceptable because of limited memory space, which is the second question.…”

Section: Resultsmentioning

confidence: 76%

“…1a). This approach captures the animals covering more view angles and helps to overcome the challenge of frequent occlusion [11][12][13][14] . The camera array is used to capture images of a checkerboard for camera calibration, followed by videos of two free-moving animals for the social behavior test (Video capture phase 1, Fig.…”

Section: Resultsmentioning

confidence: 99%

“…Firstly, the MouseVenue3D automatic camera calibration system is employed to acquire the camera parameters of the camera array (Fig. 2e left top) 11,13 . Then, based on the epipolar constraint of camera parameters, the combination of each animal instance in each camera view is optimized to achieve minimum reprojection error (Fig.…”

Section: Resultsmentioning

confidence: 99%

“…To assess the generalizability of SBeA to different animal species and experimental settings, the behaviors of birds and dogs were captured using the MouseVenue3D system with varying device configurations 11 . The animals were prepared to have as similar appearances as possible (Fig.…”

Section: Resultsmentioning

confidence: 99%

See 3 more Smart Citations

Social Behavior Atlas: A computational framework for tracking and mapping 3D close interactions of free-moving animals

Han

Chen

Wang

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

The study of social behaviors in animals is essential for understanding their survival and reproductive strategies. However, accurately tracking and analyzing the social interactions of free-moving animals has remained a challenge. Existing multi-animal pose estimation techniques suffer from drawbacks such as the need for extensive manual annotation and difficulty in discriminating between similar-looking animals in close social interactions. In this paper, we present the Social Behavior Atlas (SBeA), a novel computational framework that solves these challenges by employing a deep learning-based video instance segmentation model, 3D pose reconstruction, and unsupervised dynamic behavioral clustering. SBeA framework also involves a multi-camera setup to prevent occlusion, and a novel approach to identify individual animals in close social interactions. We demonstrate the effectiveness of SBeA in tracking and mapping the 3D close interactions of free-moving animals using the example of genetic mutant mice, birds, and dogs. Our results show that SBeA is capable of identifying subtle social interaction abnormalities, and the models and frameworks developed can be applied to a wide range of animal species. SBeA is a powerful tool for researchers in the fields of neuroscience and ecology to study animal social behaviors with a high degree of accuracy and reliability.

show abstract

Section: Resultsmentioning

confidence: 76%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

See 2 more Smart Citations

Social Behavior Atlas: A computational framework for tracking and mapping 3D close interactions of free-moving animals

Han

Chen

Wang

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Another method, Anipose [81], built on DeepLabCut, allows 3D reconstruction from multiple cameras using a wider variety of methods. MouseVenue3D [57] contributed an automated 3D behavioral capture system that uses multi-view cameras to estimate 3D pose in marker-less rodents. This work showed that 3D behavioral data yields improved performance in behavior recognition tasks compared to 2D behavioral data.…”

Section: D Animal Pose Estimationmentioning

confidence: 99%

CNN-Based Action Recognition and Pose Estimation for Classifying Animal Behavior from Videos: A Survey

Perez¹,

Toler‐Franklin²

2023

Preprint

View full text Add to dashboard Cite

Classifying the behavior of humans or animals from videos is important in biomedical fields for understanding brain function and response to stimuli. Action recognition, classifying activities performed by one or more subjects in a trimmed video, forms the basis of many of these techniques. Deep learning models for human action recognition have progressed significantly over the last decade. Recently, there is an increased interest in research that incorporates deep learning-based action recognition for animal behavior classification. However, human action recognition methods are more developed. This survey presents an overview of human action recognition and pose estimation methods that are based on convolutional neural network (CNN) architectures and have been adapted for animal behavior classification in neuroscience. Pose estimation, estimating joint positions from an image frame, is included because it is often applied before classifying animal behavior. First, we provide foundational information on algorithms that learn spatiotemporal features through 2D, two-stream, and 3D CNNs. We explore motivating factors that determine optimizers, loss functions and training procedures, and compare their performance on benchmark datasets. Next, we review animal behavior frameworks that use or build upon these methods, organized by the level of supervision they require. Our discussion is uniquely focused on the technical evolution of the underlying CNN models and their architectural adaptations (which we illustrate), rather than their usability in a neuroscience lab. We conclude by discussing open research problems, and possible research directions. Our survey is designed to be a resource for researchers developing fully unsupervised animal behavior classification systems of which there are only a few examples in the literature.CCS Concepts: • Computing methodologies → Neural networks.

show abstract

Noninvasive Tracking of Every Individual in Unmarked Mouse Groups Using Multi-Camera Fusion and Deep Learning

Wang

Wei

et al. 2022

Neurosci. Bull.

View full text Add to dashboard Cite

MouseVenue3D: A Markerless Three-Dimension Behavioral Tracking System for Matching Two-Photon Brain Imaging in Free-Moving Mice

Cited by 12 publications

References 81 publications

Social Behavior Atlas: A computational framework for tracking and mapping 3D close interactions of free-moving animals

Social Behavior Atlas: A computational framework for tracking and mapping 3D close interactions of free-moving animals

CNN-Based Action Recognition and Pose Estimation for Classifying Animal Behavior from Videos: A Survey

Noninvasive Tracking of Every Individual in Unmarked Mouse Groups Using Multi-Camera Fusion and Deep Learning

Contact Info

Product

Resources

About