DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels

Abstract

Videos of animal behavior are used to quantify researcher-defined behaviors-of-interest to study neural function, gene mutations, and pharmacological therapies. Behaviors-of-interest are often scored manually, which is time-consuming, limited to few behaviors, and variable across researchers. We created DeepEthogram: software that uses supervised machine learning to convert raw video pixels into an ethogram, the behaviors-of-interest present in each video frame. DeepEthogram is designed to be general-purpose and applicable across species, behaviors, and video-recording hardware. It uses convolutional neural networks to compute motion, extract features from motion and images, and classify features into behaviors. Behaviors are classified with above 90% accuracy on single frames in videos of mice and flies, matching expert-level human performance. DeepEthogram accurately predicts rare behaviors, requires little training data, and generalizes across subjects. A graphical interface allows beginning-to-end analysis without end-user programming. DeepEthogram's rapid, automatic, and reproducible labeling of researcher-defined behaviors-of-interest may accelerate and enhance supervised behavior analysis.

LabGym: quantification of user-defined animal behaviors using learning-based holistic assessment

“…Complex deep neural networks such as ResNet50 (He et al, 2016), which have been used in previous behavioral analysis tools (Bohnslav et al ., 2021; Mathis et al ., 2018), might not be suitable for simple larval behaviors since these complex networks would learn too many irrelevant details in simple datasets, resulting in reduced accuracy (i.e., overfitting). In contrast, LarvaN with simple complexity (Supplementary Table 1) achieved 97% accuracy in frame-wise categorizations of 6 different behaviors elicited by noxious stimuli (Figure 5A and Supplementary Video 8).…”

“…Recently, DeepEthogram has been developed to use deep neural networks to assess raw pixels of video frames for behavior classification (Bohnslav et al ., 2021), which do not rely on pre-defined features. However, since it analyzes every pixel in a video frame and thus cannot track the animal, the quantification of behavior, such as the measurements on the body motion magnitude or speed during a behavior, is missing.…”

“…It assesses all pixels of an entire video frame, even when generating the optical flow of animal motion. In fact, the noise in the optical flows caused by irrelevant background can be clearly visualized in DeepEthogram (Bohnslav et al ., 2021). It also uses the complex, fixed ResNet (He et al ., 2016) structures, which might not work well for simple behavior data such as larval behaviors and makes the training procedures complicated.…”

“…Although transfer learning that applies acquired network weights to new categorization problems can potentially be used to compensate for the lack of species-specific behavioral datasets, its efficiency relies on the similarity between the existing datasets used for pretraining the networks and the new data (Yosinski et al, 2014). Current tools using transfer learning for behavioral categorizations (Bohnslav et al ., 2021) utilized network weights that were acquired from human behavioral datasets such as Kinetics-700 (Carreira et al, 2019). However, human behaviors are intrinsically different from the behaviors of model organisms, especially the soft-bodied ones.…”

“…It then uses another convolutional neural network to evaluate a ‘pattern image’ that we introduced, which is a 2-dimensional imprint of animal’s overall movement pattern during the episode. This approach ignores irrelevant background and only focuses on the whole body and body parts of behaving animal and is much more efficient and accurate than current approaches for estimating motion flows using all pixels (Bohnslav et al ., 2021; Wu et al, 2020). Together with other technical improvements, including a new method for effective background subtraction that outperforms the state-of-art technologies, this new neural network made LabGym run easily on a common laptop computer while most current deep-learning-based tools for behavioral analysis require demanding computational capabilities (Bohnslav et al ., 2021; Mathis et al ., 2018).…”