Fast animal pose estimation using deep neural networks

Abstract

Recent work quantifying postural dynamics has attempted to define the repertoire of behaviors across the entire data set, we use a technique we refer to as cluster sampling. A simple random 117 subset of the movie images are grouped via k-means clustering and then these images are 118 sampled uniformly across groups for labeling. The grouping is based on linear correlations 119 between pixel intensities in the images as a proxy measure for similarity in body pose. The 120 diversity of poses represented using this method can be observed in the centroids of each of the 121 clusters identified (Supplementary Fig. 2). 122 123 Poses in each training image are labeled using a custom GUI with draggable body part markers 124 that form a skeleton (Fig. 1b). For the fruit fly, we track four points on each of the six legs, two 125 points on the wing tips, three points on the thorax and abdomen, and three points on the head 126 for a total of 32 points in every frame. These points were chosen to align with known Drosophila 127 body joints (Supplementary Fig. 3). For every training image, the user drags each skeleton 128 point to the appropriate body part and the program saves the label positions into a self-129 contained file. To enhance the size of the training image set further without the need for hand 130 labeling more frames, we augment the dataset by applying small random rotations and body-131 axis reflections to generate new samples from the labeled data. As the neural network 132 processes the raw images, the rotated and reflected images add new information that the 133 network can use during training. 134 135

Markerless tracking of an entire insect colony

“…Recent machine vision advances in the precise posture tracking of individual animals 17,18,52 as well as of the positions of highly-similar organisms in groups 46 are enabling new quantitative studies of behavior 53,54 . In collective behavior specifically, the use of CNNs for the pixel-based identification of individual organisms has significantly advanced markerless, long-time tracking in 2D, from more modest assemblies (~10 individuals) 34 to large groups (~100 individuals) 46 .…”

“…Recent breakthroughs in image analysis using CNNs, including fast and accurate single and multiple object detection [12][13][14] , posture quantification [15][16][17][18] and image and video appearance representation [19][20][21] , offer new inspiration and opportunities for extracting information directly from video data in denseanimal contexts. However, most existing solutions and benchmark datasets for multi-object tracking as well as for posture and activity recognition, are dedicated to human behavior and crowds [22][23][24][25] .…”

“…In biological image data CNNs have been broadly exploited for cell or particle segmentation [26][27][28][29] . Versatile, supervised CNN-based tools have also been proposed for animal posture quantification 17,18 and have been successfully applied to the study of insect behavior [30][31][32] . While facilitating important tasks in bioimage interpretation, these solutions are limited to behavior of few individuals and do not resolve challenges within the task of dense object detection and tracking in a bee colony.…”

A low-cost, open-source framework for tracking and behavioural analysis of animals in aquatic ecosystems

“…Coastal regions in both marine and freshwater 35 environments are also those that are of greatest interest for eco-tourism, community 36 fisheries, and industry, while simultaneously being most affected by habitat degradation, 37 exploitation, and anthropogenic pollution. 38 Yet aquatic ecosystems appear to be poorly represented in movement ecology 39 research with only nine publications in the journal Movement Ecology containing one of 40 the following keywords in the title: 'ocean', 'aquatic', 'littoral', 'marine', 'fish', 'sea' [21]. 41 Moreover, of the research into marine or freshwater animal movement, there is a heavy 42 bias towards larger marine fauna, which often inhabit open oceanic areas [21].…”

“…311 In addition, 3D pose estimation is now possible for wild animals, enabling exact 312 reconstruction of the entire animal [54]. There has been a shift in how animal movement 313 is analyzed in light of computational ethological approaches [5,39,42], with patterns of 314 motion able to be objectively disentangled, revealing the underlying behavioural syntax 315 to the observer. Automated approaches based on video, or even audio, recordings may 316 also overcome sensory limitations of human systems, allowing a better understanding of 317 the sensory umwelt of study species [25] and also facilitate novel experimental 318 designs [2,42] that can tackle questions of the proximate and ultimate causality of 319 behaviour [5,39,54].…”