DeepFly3D: A deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila

Abstract

Studying how neural circuits orchestrate limbed behaviors requires the precise 10 measurement of pose-the positions of each appendage-in 3-dimensional (3D) space. Recent 11 advances in computer vision and machine learning have made it possible to use deep neural 12 networks to estimate 2-dimensional (2D) pose in freely behaving and tethered animals. However, 13the unique challenges associated with transforming these measurements into reliable and precise 14 3D poses have not been addressed for small animals including the fly, Drosophila melanogaster. 15 Here we present DeepFly3D, a computational pipeline for inferring the 3D pose of tethered, adult 16 Drosophila using multiple camera images. First, we introduce an approach for multiple-camera 17 calibration using the animal itself rather than the typical checkerboard or similar external 18 apparatus. Second, we present an iterative approach that robustly infers 3D pose using graphical 19 models and deep-learning based 2D predictions from multiple cameras. False predictions are 20 rejected using an optimization scheme based on dynamic programming and belief propagation. To 21 close the loop, the corrected poses are used to retrain the 2D pose deep estimation network and to 22 improve 3D pose estimation. Finally, we provide a graphical user interface (GUI) and active learning 23 policy for interacting with, annotating, and correcting 3D pose data. Emphasizing the importance of 24 our tool, we demonstrate that unsupervised behavioral embedding of 3D joint angles yields more 25 accurate behavioral maps than those generated with 2D pose data because the latter are highly 26 perspective-dependent. We provide our DeepFly3D deep network and weights, training data, 27 computational pipeline, and nearly one million images of tethered, behaving flies along with 28 corresponding 3D joint positions 1 . These tools make it possible to acquire high-fidelity behavioral 29 measurements at an unprecedented level of precision and resolution for a variety of biological 30 applications. 31 32 35 exoskeletons, these measurements are naturally made with reference to 3D joint and appendage 36 locations. Paired with modern approaches to simultaneously record the activity of neural popu-37 lations in tethered, behaving animals (Dombeck et al., 2007; Seelig et al., 2010; Chen et al., 2018), 38 3D joint and appendage tracking promises to accelerate the dissection of neural control principles, 39 1 https://github.com/NeLy-EPFL/DeepFly3D particularly in the genetically tractable and numerically simple nervous system of the fly, Drosophila 40 melanogaster. 41 However, algorithms for reliably estimating 3D pose in such small animals have not yet been 42 developed. Instead, multiple alternative approaches have been taken. For example, one can affix 43 and use small markers-reflective, colored, or fluorescent particles-to identify and reconstruct 44 keypoints from video data (Bender et al., 2010; Kain et al., 2013; Todd et al., 2017). Although this 45 approach works well on h...

Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio

“…The ability to track positions of macaques is important because of their central role in biomedical research, as well as their importance in psychology, and ethology. Recent years have witnessed the development of widely used markerless tracking systems in many species, including flies, worms, mice and rats, and humans 16 , 20 , 32 , 33 . Such systems are typically not designed with the specific problems of monkey pose estimation in mind.…”

“…That is, the major barrier to successful pose tracking in macaques is the lack of a sufficiently large reliably annotated training set, rather than the lack of an algorithm that can estimate pose given that set. A similar approach has recently been successfully applied in the tracking of flies constrained by tethers 20 on which our work naturally builds and extends to generalized free movement.…”

“…Methods that make use of transfer learning work for various organisms like flies, worms, and mice by learning from a larger number of images collected from a single view 16 – 19 . Two extremely promising extensions towards view-invariant pose estimation in animal models have recently been demonstrated in cheetahs 17 and in flies by integrating active learning 20 .…”

Fully Automated Leg Tracking in Freely Moving Insects using Feature Learning Leg Segmentation and Tracking (FLLIT)

“…There are now a number of methods for leg movement tracking of Drosophila and other animals 1,2,3,4,14,15,16 , giving researchers a wide range of options depending on the goals of the experiment. Some of these are foot printing-based approaches, which are highly accurate but which report only claw contact points with the detection surface 4,14…”

“…. On the other hand, recent deep learning approaches 2,3,16 are highly versatile, allowing analysis of behaviors that require tracking of leg joints and other body parts in any animal, with the caveat that the algorithms need to first be trained with user annotated datasets. A third type of approach uses morphology or image-contrast-based methods 1,15,17 to find the outline of each leg to identify claw positions.…”