Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture From Images “In the Wild”

Abstract

Figure 1: Zebras from images. We automatically extract 3D textured models of zebras from in-the-wild images. We regress directly from pixels, without keypoint detection or segmentation. AbstractWe present the first method to perform automatic 3D pose, shape and texture capture of animals from images acquired in-the-wild. In particular, we focus on the problem of capturing 3D information about Grevy's zebras from a collection of images. The Grevy's zebra is one of the most endangered species in Africa, with only a few thousand individuals left. Capturing the shape and pose of these animals can provide biologists and conservationists with information about animal health and behavior. In contrast to research on human pose, shape and texture estimation, training data for endangered species is limited, the animals are in complex natural scenes with occlusion, they are naturally camouflaged, travel in herds, and look similar to each other. To overcome these challenges, we integrate the recent SMAL animal model into a networkbased regression pipeline, which we train end-to-end on synthetically generated images with pose, shape, and background variation. Going beyond state-of-the-art methods for human shape and pose estimation, our method learns a shape space for zebras during training. Learning such a shape space from images using only a photometric loss is novel, and the approach can be used to learn shape in other settings with limited 3D supervision. Moreover, we couple 3D pose and shape prediction with the task of texture synthesis, obtaining a full texture map of the animal from a single image. We show that the predicted texture map allows a novel per-instance unsupervised optimization over the network features. This method, SMALST (SMAL with learned Shape and Texture) goes beyond previous work, which assumed manual keypoints and/or segmentation, to regress directly from pixels to 3D animal shape, pose and texture. Code and data are available at

Deep learning tools for the measurement of animal behavior in neuroscience

“…Typically, large datasets are collected to enable the creation of robust algorithms for inference on diverse humans (or for animals, scanning toy models has been fruitful (44)). Recently, outstanding improvements have been made to capture shapes of animals from images (22,45,46). However, there are no animal-specific toolboxes geared towards neuroscience applications, although we believe that this will change in the near future, as for many applications having the soft-tissue measured will be highly important, i.e.…”

DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning

“…Future advances will likely allow for the calibration and synchronizaton of imaging devices across multiple UAVs (e.g., Price et al, 2018; Saini et al, 2019). This would make it possible to measure the full 3-D posture of wild animals (e.g., Zuffi et al, 2019) in scenarios where fixed camera systems (e.g., Nath et al, 2019) would not be tractable, such as during migratory or predation events. When combined, these technologies could allow researchers to address questions about the behavioral ecology of animals that were previously impossible to answer.…”

“…New pose estimation methods are already replacing human annotations with fully articulated volumetric 3-D models of the animal’s body (e.g., the SMAL model from Zuffi et al, 2017 or the SMALST model from Zuffi et al, 2019), and the 3-D scene can be estimated using unsupervised, semi-supervised, or weakly-supervised methods (e.g., Jaques et al, 2019; Zuffi et al, 2019), where the shape, position, and posture of the animal’s body, the camera position and lens parameters, and the background environment and lighting conditions are jointly learned directly from 2-D images by a deep-learning model (Valentin et al, 2019; Zuffi et al, 2019). These inverse graphics models (Kulkarni et al, 2015; Sabour et al, 2017; Valentin et al, 2019) take advantage of recently developed differentiable graphics engines that allow 3-D rendering parameters to be controlled using standard optimization methods (Zuffi et al, 2019; Valentin et al, 2019). After optimization, the volumetric 3-D timeseries data predicted by the deep learning model could be used directly for behavioral analysis or specific keypoints or body parts could be selected for analysis post-hoc.…”

MacaquePose: A novel ‘in the wild’ macaque monkey pose dataset for markerless motion capture

“…The performance evaluation of the network trained with the present dataset revealed that there is still room for improvement regarding the misattribution of the limb keypoints ( Figure 2h, Table 2), although the RMSE indicates the human-level performance (Figure 3). The DeepLabCut algorithm (Mathis et al, 2018) used in the present evaluation does not explicitly utilize the prior knowledge about the animal's body, whereas the other algorithms were suggested to use the connection between keypoints (Insafutdinov et al, 2016;Cao et al, 2017) or 3D shape of the subject (Biggs et al, 2018;Zuffi et al, 2019). Such utilization of the prior knowledge may help to improve the estimation.…”