Limbs Detection and Tracking of Head-Fixed Mice for Behavioral Phenotyping Using Motion Tubes and Deep Learning

Abstract

The broad accessibility of affordable and reliable recording equipment and its relative ease of use has enabled neuroscientists to record large amounts of neurophysiological and behavioral data. Given that most of this raw data is unlabeled, great effort is required to adapt it for behavioral phenotyping or signal extraction, for behavioral and neurophysiological data, respectively. Traditional methods for labeling datasets rely on human annotators which is a resource and time intensive process, which often produce data that that is prone to reproducibility errors. Here, we propose a deep learning-based image segmentation framework to automatically extract and label limb movements from movies capturing frontal and lateral views of head-fixed mice. The method decomposes the image into elemental regions (superpixels) with similar appearance and concordant dynamics and stacks them following their partial temporal trajectory. These 3D descriptors (referred as motion cues) are used to train a deep convolutional neural network (CNN). We use the features extracted at the last fully connected layer of the network for training a Long Short Term Memory (LSTM) network that introduces spatio-temporal coherence to the limb segmentation. We tested the pipeline in two video acquisition settings. In the first, the camera is installed on the right side of the mouse (lateral setting). In the second, the camera is installed facing the mouse directly (frontal setting). We also investigated the effect of the noise present in the videos and the amount of training data needed, and we found that reducing the number of training samples does not result in a drop of more than 5% in detection accuracy even when as little as 10% of the available data is used for training.

Research on Three-Dimensional Pose Reconstruction of Experimental Mice with Depth Information Fusion

“…Where f x and f y are used instead of af and bf , and f x and f y are measured in pixels. Using homogeneous coordinates, equation ( 13) can be written in matrix form as shown in equation (14).…”

“…Later methods involved implanting sensor chips or placing markers on mice to detect multiple key body points [10,11]. With the advent of deep learning, convolutional neural networks have been employed for the unmarked detection of whole-body semantic feature keypoints based on mouse anatomical structures [12][13][14][15].However, 2D analysis has limitations. It struggles with recognizing behaviors in the vertical dimension [16], cannot intuitively represent mice's movements in the three-dimensional world, and different three-dimensional poses might correspond to the same 2D pose.…”

Statistical signature of subtle behavioural changes in large-scale behavioural assays

“…Advances in machine learning [11,12] have recently complemented automated behavioural analyses, and supervised [13][14][15][16][17][18][19][20] and unsupervised methods [21][22][23][24][25][26][27][28][29] have been introduced alongside image feature-based approaches to identify behaviours. Some methods can be applied broadly to various experiments after an annotation phase, like DeepLabCut [13], or, as with some unsupervised approaches, others are more specialised and apply to one animal in a specified behavioural paradigm [30,31]. Supervised techniques aim to define behaviours based on external expertise, while unsupervised ones seek to have them naturally emerge, later undergoing post-hoc validation by experts.…”

Non-Consummatory Behavior Signals Predict Aversion-Resistant Alcohol Drinking in Head-Fixed Mice

“…For instance, these techniques make it possible to record from large portions of the cortical surface using widefield calcium imaging [42], to simultaneously record from and manipulate brain region level populations with two-photon calcium imaging and optogenetics [43], or record from thousands of neurons simultaneously across multiple brain regions using multiple high-density electrophysiology probes [44]. Furthermore, due to the stable positioning of the animal, these methods facilitate motion tracking of individual body parts [45] and advanced behavioral analyses [46][47][48][49][50][51][52][53][54][55], as well as the precise delivery of visual [56], auditory [57], and olfactory stimuli [58,59]. This improved precision reduces noise between behavioral data, stimuli, and neural activity in comparison to freely moving mice.…”

“…Head-fixed mouse preparations offer the potential to meet this need because these methods allow researchers to exploit powerful neural recording techniques, such as calcium imaging [31,32] and high-yield, multi-brain region electrophysiology [33,34]. Furthermore, due to the stable positioning of the animal, these methods facilitate motion tracking of individual body parts [35] and advanced behavioral analyses [36][37][38][39][40][41][42][43][44][45]. Head-fixed methods have been widely utilized in various types of non-addiction related studies [46][47][48].…”