Real-time markerless video tracking of body parts in mice using deep neural networks

Abstract

Markerless and accurate tracking of mouse movement is of interest to many biomedical, pharmaceutical, and behavioral science applications. The additional capability of tracking body parts in real-time with minimal latency opens up the possibility of manipulating motor feedback, allowing detailed explorations of the neural basis for behavioral control. Here we describe a system capable of tracking specific movements in mice at a frame rate of 30.3 Hz. To achieve these results, we adapt DeepLabCut – a robust movement-tracking deep neural network framework – for real-time tracking of body movements in mice. We estimate paw movements of mice in real time and demonstrate the concept of movement-triggered optogenetic stimulation by flashing a USB-CGPIO controlled LED that is triggered when real time analysis of movement exceeds a pre-set threshold. The mean time delay between movement initiation and LED flash was 93.44 ms, a latency sufficient for applying behaviorally-triggered feedback. This manuscript presents the rationale and details of the algorithms employed and shows implementation of the system using behaving mice. This system lays the groundwork for a behavior-triggered ‘closed loop’ brain-machine interface with optogenetic stimulation of specific brain regions for feedback.

Deep-learning based identification, tracking, pose estimation, and behavior classification of interacting primates and mice in complex environments

“…We use a recognition network to extract features from the analysis of single frames, that we base on the Xception 21 network architecture. We initialize parts of the network with ImageNet 4 weights. These features are then integrated over time by a TCN 22,23 to classify the behavior of the animal in each frame (see Methods for architecture and training details).…”

Track-Control, an automatic video-based real-time closed-loop behavioral control toolbox

“…With the rapid development of computer vision and machine learning, object detection has welcomed its great progress and enabled us to automatically analyze videos, increasing the objectivity. For example, DeepLabCut has successfully achieved markerless feature extraction from pre-recorded videos (Mathis et al, 2018;Nath et al, 2019), and another group has demonstrated that the method has potential of being applied in real-time feedback control (Forys et al, 2018). DeepLabCut was built upon ResNet and not focused on the real-time detection.…”

Bayesian Multi-Targets Strategy to Track Apis mellifera Movements at Colony Level

“…However, it is still a challenge to solve complex problems, such as tracking similar objects near each other [23,24], especially under heavy background noise [18,23]. Using deep neural networks has provided significant contributions [25], but these approaches are still computationally expensive, hindering their widespread application [26]. Recent approaches to object tracking (e.g., hierarchical learned features for tracking and cognitive vision) present similar difficulties as algorithms and are limited to tracking only a few objects at the same time [27,28].…”