The integration of sensorimotor signals to internally estimate self-movement is critical for spatial perception and motor control. However, which neural circuits accurately track body motion and how these circuits control movement remain unknown. We found that a population of Drosophila neurons that were sensitive to visual flow patterns typically generated during locomotion, the horizontal system (HS) cells, encoded unambiguous quantitative information about the fly's walking behavior independently of vision. Angular and translational velocity signals were integrated with a behavioral-state signal and generated direction-selective and speed-sensitive graded changes in the membrane potential of these non-spiking cells. The nonvisual direction selectivity of HS cells cooperated with their visual selectivity only when the visual input matched that expected from the fly's movements, thereby revealing a circuit for internally monitoring voluntary walking. Furthermore, given that HS cells promoted leg-based turning, the activity of these cells could be used to control forward walking.

Videos of animal behavior are used to quantify researcher-defined behaviors-of-interest to study neural function, gene mutations, and pharmacological therapies. Behaviors-of-interest are often scored manually, which is time-consuming, limited to few behaviors, and variable across researchers. We created DeepEthogram: software that uses supervised machine learning to convert raw video pixels into an ethogram, the behaviors-of-interest present in each video frame. DeepEthogram is designed to be general-purpose and applicable across species, behaviors, and video-recording hardware. It uses convolutional neural networks to compute motion, extract features from motion and images, and classify features into behaviors. Behaviors are classified with above 90% accuracy on single frames in videos of mice and flies, matching expert-level human performance. DeepEthogram accurately predicts rare behaviors, requires little training data, and generalizes across subjects. A graphical interface allows beginning-to-end analysis without end-user programming. DeepEthogram's rapid, automatic, and reproducible labeling of researcher-defined behaviors-of-interest may accelerate and enhance supervised behavior analysis.

Researchers commonly acquire videos of animal behavior and quantify the prevalence of behaviors of interest to study nervous system function, the effects of gene mutations, and the efficacy of pharmacological therapies. This analysis is typically performed manually and is therefore immensely time consuming, often limited to a small number of behaviors, and variable across researchers. Here, we created DeepEthogram: software that takes raw pixel values of videos as input and uses machine learning to output an ethogram, the set of user-defined behaviors of interest present in each frame of a video. We used convolutional neural network models that compute motion in a video, extract features from motion and single frames, and classify these features into behaviors. These models classified behaviors with greater than 90% accuracy on single frames in videos of flies and mice, matching expert-level human performance. The models accurately predicted even extremely rare behaviors, required little training data, and generalized to new videos and subjects. DeepEthogram runs rapidly on common scientific computer hardware and has a graphical user interface that does not require programming by the end-user. We anticipate DeepEthogram will enable the rapid, automated, and reproducible assignment of behavior labels to every frame of a video, thus accelerating all those studies that quantify behaviors of interest.Code is available at: https://github.com/jbohnslav/deepethogram