3D Menagerie: Modeling the 3D Shape and Pose of Animals

Abstract

Figure 1: Animals from images. We learn an articulated, 3D, statistical shape model of animals using very little training data. We fit the shape and pose of the model to 2D image cues showing how it generalizes to previously unseen shapes. AbstractThere has been significant work on learning realistic, articulated, 3D models of the human body. In contrast, there are few such models of animals, despite many applications. The main challenge is that animals are much less cooperative than humans. The best human body models are learned from thousands of 3D scans of people in specific poses, which is infeasible with live animals. Consequently, we learn our model from a small set of 3D scans of toy figurines in arbitrary poses. We employ a novel partbased shape model to compute an initial registration to the scans. We then normalize their pose, learn a statistical shape model, and refine the registrations and the model together. In this way, we accurately align animal scans from different quadruped families with very different shapes and poses. With the registration to a common template we learn a shape space representing animals including lions, cats, dogs, horses, cows and hippos. Animal shapes can be sampled from the model, posed, animated, and fit to data. We demonstrate generalization by fitting it to images of real animals including species not seen in training.

Deep learning tools for the measurement of animal behavior in neuroscience

“…However, state-of-the-art performance currently requires body-scanning of many subjects to make body models. Typically, large datasets are collected to enable the creation of robust algorithms for inference on diverse humans (or for animals, scanning toy models has been fruitful (44)). Recently, outstanding improvements have been made to capture shapes of animals from images (22,45,46).…”

“…This is a difficult challenge as zebras are designed to blend into the background in the safari. This paper makes significant improvements on accuracy and realism, and builds on a line of elegant work from these authors (44,46) [6**] DeeperCut: A deeper, stronger, and faster multiperson pose estimation model (29) DeeperCut is a highly accurate algorithm for multi-human pose estimation due to improved deep learning based body part detectors, and image-conditioned pairwise terms to predict the location of body parts based on the location of other body parts. These terms are then used to find accurate poses of individuals via graph cutting.…”

Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture From Images “In the Wild”

“…For comparison, we also perform per-instance optimization over the model variables (D). [34], which requires ground truth keypoints and segmentations; (B) We run the network feed-forward prediction on a synthetic dataset; (C) our proposed method; (D) per-instance optimization on model variables rather than network features; (F) feed-forward prediction (no optimization); (G) feed-forward prediction without texture; (H) feed-forward prediction with noise on the bounding boxes.…”

“…The SMAL model is a function M (β, θ, γ) of shape β, pose θ and translation γ. β is a vector of the coefficients of the learned PCA shape space, θ ∈ R 3N = {r i } N i=1 is the relative rotation, expressed with Rodrigues vectors, of the joints in the kinematic tree, and γ is the global translation applied to the root joint. Unlike [34], which uses N = 33 joints, we segment and add articulation to the ears, obtaining a model with N = 35 body parts. The SMAL function returns a 3D mesh, where the model template is shaped by β, articulated by θ and shifted by γ.…”

Geometric Disentanglement for Generative Latent Shape Models

“…For our experiments, we consider four datasets of meshes: shapes computed from the MNIST dataset [37], the MPI Dyna dataset of human shapes [49], a dataset of animal shapes from the Skinned Multi-Animal Linear model (SMAL) [65], and a dataset of human shapes from the Skinned Multi-Person Linear model (SMPL) [40] via the SURREAL dataset [60]. For each, we generate point clouds of size N T via area-weighted sampling.…”