DeepLabCut: markerless pose estimation of user-defined body parts with deep learning

Abstract

Quantifying behavior is crucial for many applications in neuroscience. Videography provides easy methods for the observation and recording of animal behavior in diverse settings, yet extracting particular aspects of a behavior for further analysis can be highly time consuming. In motor control studies, humans or other animals are often marked with reflective markers to assist with computer-based tracking, but markers are intrusive, and the number and location of the markers must be determined a priori. Here we present an efficient method for markerless pose estimation based on transfer learning with deep neural networks that achieves excellent results with minimal training data. We demonstrate the versatility of this framework by tracking various body parts in multiple species across a broad collection of behaviors. Remarkably, even when only a small number of frames are labeled (~200), the algorithm achieves excellent tracking performance on test frames that is comparable to human accuracy.

Two brain pathways initiate distinct forward walking programs inDrosophila

“…S4B). This step of the annotation was done automatically in DeepLabCut (DLC, Mathis et al, 2018); to use this approach we trained and evaluated DLC with a data set of 1000 manually annotated video frames (10 flies, 100 exemplary frames each), that were similar to the ones we recorded during the experiments described here. One half (500 frames, 10 flies, 50 frames each) of this set was used for training DLC, the other half was used to evaluate its performance.…”

The Object Space Task reveals a dissociation between semantic-like and episodic-like memory in a mouse model of Kleefstra Syndrome

“…On a per-session level including all 21 trials, we have shown that Ehmt1 +/mice explore objects less and show increased memory expression, which is specific to our semantic-like memory condition. Next, we analyzed discrete behaviors on a trial-by-trial level: We trained two classifiers on multiple behavioral features extracted from the video data automatically with deep-learning methods as explained in the Materials and Methods section 18 . We included both general exploration features (e.g.…”

“…Automatic classification of genotype based on video analysis is becoming more popular and shows great potential for monitoring treatment outcomes in pre-clinical studies. Both 3D and 2D video techniques can be used 18,19 . Our findings highlight the importance of considering which behaviors are recorded with the video-data: Behaviors with higher cognitive demands (such as our overlapping condition) may be more sensitive to genotype differences as we can show here.…”

“…Transfer learning was applied to a restructured version of the neural network and re-trained on videos of mice performing the Object Space Task, labelled by humans for object exploration. Next, the second model uses DeepLabCut 18 to extract limb configuration of mice over a single video. In addition to limb configuration, head direction was calculated as ears pointing to the nose.…”

“…The task contains both an episodic-like as well as semantic-like memory condition. In addition to the conventional behavioral analysis, we modeled the learning behavior over training days as well as build a classifier for individual trial behaviors automatically extracted from the videos with in-house deep learning networks based on DeepLabCut 18 . Ehmt1 +/mice showed overall decreased exploration of the objects and more corner sitting but expressed increased semantic-like memory compared to controls.…”