C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion

Abstract

Factorization network Te s t Train ϕ 3D shape and viewpoint (α, θ ) (Y, v) 2D keypoints and visibility Dense keypoints Non-rigid objects Rigid objects Monocular reconstruction of:ϕ Figure 1: Our method learns a 3D model of a deformable object category from 2D keypoints in unconstrained images. It comprises a deep network that learns to factorize shape and viewpoint and, at test time, performs monocular reconstruction. AbstractWe propose C3DPO, a method for extracting 3D models of deformable objects from 2D keypoint annotations in unconstrained images. We do so by learning a deep network that reconstructs a 3D object from a single view at a time, accounting for partial occlusions, and explicitly factoring the effects of viewpoint changes and object deformations. In order to achieve this factorization, we introduce a novel regularization technique. We first show that the factorization is successful if, and only if, there exists a certain canonicalization function of the reconstructed shapes. Then, we learn the canonicalization function together with the reconstruction one, which constrains the result to be consistent. We demonstrate stateof-the-art reconstruction results for methods that do not use ground-truth 3D supervision for a number of benchmarks, including Up3D and PASCAL3D+. Source code has been made available at https

Animals in Virtual Environments

“…S2,S3 in supplementary). Moreover, extracting 3D posture of animals from images and videos is an emerging topic in the computer vision community [37,55,63,88]. Posture based video analysis and activity recognition are currently being investigated using deep learning techniques.…”

Unsupervised Learning of Category-Specific Symmetric 3D Keypoints from Point Sets

“…Linear low-rank shape basis [2,34,35], low-rank trajectory basis [36], isometry or piece-wise rigidity [37,38] are some of the different methods used for NRSfM. Recently, a few number of works have used low-rank shape basis in order to devise learned methods [1,31,33,39]. Another useful tool in modeling shape category is the reflective symmetry, which is also directly related to the object pose.…”

“…Several recent works have modeled shapes in a category as instances of nonrigid deformations [1,31,33,39]. The motivation lies in the fact that such shapes often share similarities to a large extent.…”

“…In fact, keypoints-based methods have been crucial to the success of many vision applications. A few examples include; 3D reconstruction [1][2][3], registration [4][5][6][7], human body pose [8][9][10][11], recognition [12,13], and generation [14,15]. That being said, many keypoints are defined manually, while considering their semantic locations such as facial landmarks and human body joints, to serve and simplify the problem at hand.…”

Automatic mapping of multiplexed social receptive fields by deep learning and GPU-accelerated 3D videography

“…This makes it easy to update the algorithm with methodo-492 logical advances in the field. Deep learning in 3D is still thought to be in its infancy 68 , but along with technical 493 developments in depth imaging hardware (for video games, self-driving cars and other industrial applications), 494 there are exciting developments in analysis, including deep leaning methods for detection of deformable ob-495 jects from image [69][70][71] and point-cloud 72,73 data, geometric and graph-based tricks for GPU-accelerated analysis 496 of 3D data 74-76 , and methods for physical modeling of deformable bodies 77-79 . 497…”