Studying how neural circuits orchestrate limbed behaviors requires the precise measurement of the positions of each appendage in three-dimensional (3D) space. Deep neural networks can estimate two-dimensional (2D) pose in freely behaving and tethered animals. However, the unique challenges associated with transforming these 2D measurements into reliable and precise 3D poses have not been addressed for small animals including the fly, Drosophila melanogaster. Here, we present DeepFly3D, a software that infers the 3D pose of tethered, adult Drosophila using multiple camera images. DeepFly3D does not require manual calibration, uses pictorial structures to automatically detect and correct pose estimation errors, and uses active learning to iteratively improve performance. We demonstrate more accurate unsupervised behavioral embedding using 3D joint angles rather than commonly used 2D pose data. Thus, DeepFly3D enables the automated acquisition of Drosophila behavioral measurements at an unprecedented level of detail for a variety of biological applications.
Studying how neural circuits orchestrate limbed behaviors requires the precise 10 measurement of pose-the positions of each appendage-in 3-dimensional (3D) space. Recent 11 advances in computer vision and machine learning have made it possible to use deep neural 12 networks to estimate 2-dimensional (2D) pose in freely behaving and tethered animals. However, 13the unique challenges associated with transforming these measurements into reliable and precise 14 3D poses have not been addressed for small animals including the fly, Drosophila melanogaster. 15 Here we present DeepFly3D, a computational pipeline for inferring the 3D pose of tethered, adult 16 Drosophila using multiple camera images. First, we introduce an approach for multiple-camera 17 calibration using the animal itself rather than the typical checkerboard or similar external 18 apparatus. Second, we present an iterative approach that robustly infers 3D pose using graphical 19 models and deep-learning based 2D predictions from multiple cameras. False predictions are 20 rejected using an optimization scheme based on dynamic programming and belief propagation. To 21 close the loop, the corrected poses are used to retrain the 2D pose deep estimation network and to 22 improve 3D pose estimation. Finally, we provide a graphical user interface (GUI) and active learning 23 policy for interacting with, annotating, and correcting 3D pose data. Emphasizing the importance of 24 our tool, we demonstrate that unsupervised behavioral embedding of 3D joint angles yields more 25 accurate behavioral maps than those generated with 2D pose data because the latter are highly 26 perspective-dependent. We provide our DeepFly3D deep network and weights, training data, 27 computational pipeline, and nearly one million images of tethered, behaving flies along with 28 corresponding 3D joint positions 1 . These tools make it possible to acquire high-fidelity behavioral 29 measurements at an unprecedented level of precision and resolution for a variety of biological 30 applications. 31 32 35 exoskeletons, these measurements are naturally made with reference to 3D joint and appendage 36 locations. Paired with modern approaches to simultaneously record the activity of neural popu-37 lations in tethered, behaving animals (Dombeck et al., 2007; Seelig et al., 2010; Chen et al., 2018), 38 3D joint and appendage tracking promises to accelerate the dissection of neural control principles, 39 1 https://github.com/NeLy-EPFL/DeepFly3D particularly in the genetically tractable and numerically simple nervous system of the fly, Drosophila 40 melanogaster. 41 However, algorithms for reliably estimating 3D pose in such small animals have not yet been 42 developed. Instead, multiple alternative approaches have been taken. For example, one can affix 43 and use small markers-reflective, colored, or fluorescent particles-to identify and reconstruct 44 keypoints from video data (Bender et al., 2010; Kain et al., 2013; Todd et al., 2017). Although this 45 approach works well on h...
Knowledge of one's own behavioral state---whether one is walking, grooming, or resting---is critical for contextualizing sensory cues including interpreting visual motion and tracking odor sources. Additionally, awareness of one's own posture is important to avoid initiating destabilizing or physically impossible actions. Ascending neurons (ANs), interneurons in the vertebrate spinal cord or insect ventral nerve cord (VNC) that project to the brain, may provide such high-fidelity behavioral state signals. However, little is known about what ANs encode and where they convey signals in any brain. To address this gap, we performed a large-scale functional screen of AN movement encoding, brain targeting, and motor system patterning in the adult fly, Drosophila melanogaster. Using a new library of AN sparse driver lines, we measured the functional properties of 247 genetically-identifiable ANs by performing two-photon microscopy recordings of neural activity in tethered, behaving flies. Quantitative, deep network-based neural and behavioral analyses revealed that ANs nearly exclusively encode high-level behaviors---primarily walking as well as resting and grooming---rather than low-level joint or limb movements. ANs that convey self-motion---resting, walking, and responses to gust-like puff stimuli---project to the brain's anterior ventrolateral protocerebrum (AVLP), a multimodal, integrative sensory hub, while those that encode discrete actions---eye grooming, turning, and proboscis extension---project to the brain's gnathal ganglion (GNG), a locus for action selection. The structure and polarity of AN projections within the VNC are predictive of their functional encoding and imply that ANs participate in motor computations while also relaying state signals to the brain. Illustrative of this are ANs that temporally integrate proboscis extensions over tens-of-seconds, likely through recurrent interconnectivity. Thus, in line with long-held theoretical predictions, ascending populations convey high-level behavioral state signals almost exclusively to brain regions implicated in sensory feature contextualization and action selection.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.