End-to-End 6-DoF Object Pose Estimation Through Differentiable Rasterization

Abstract

Here we introduce an approximated differentiable renderer to refine a 6-DoF pose prediction using only 2D alignment information. To this end, a two-branched convolutional encoder network is employed to jointly estimate the object class and its 6-DoF pose in the scene. We then propose a new formulation of an approximated differentiable renderer to re-project the 3D object on the image according to its predicted pose; in this way the alignment error between the observed and the re-projected object silhouette can be measured. Since the renderer is differentiable, it is possible to back-propagate through it to correct the estimated pose at test time in an online learning fashion. Eventually we show how to leverage the classification branch to profitably re-project a representative model of the predicted class (i.e. a medoid) instead. Each object in the scene is processed independently and novel viewpoints in which both objects arrangement and mutual pose are preserved can be rendered.

Warp and Learn: Novel Views Generation for Vehicles and Other Objects

“…In particular, in Zhu et al [79] the 2.5D sketch consists of both a silhouette and a depth image rendered from a learnt low-resolution voxel grid by means of a differentiable ray-tracer. While this method is appealing for its geometrical guarantees, it is limited by a number of factors: i) it requires a custom differentiable ray-tracing module; ii) footprint of voxel-based representations scales with the cube of the resolution despite most of the information lying on the surface [54], [41]; iii) errors in the 3D voxel grid naturally propagate to the 2.5D sketch. We also follow this line of work to provide soft 3D priors to the synthesis process.…”

LaLaLoc: Latent Layout Localisation in Dynamic, Unvisited Environments

“…We propose a direct pose optimisation through differentiable rendering. While differentiable rendering-based approaches have been shown to be effective for pose estimation [24,10], these works rely on homogeneous data to compute losses between the prediction and the target, often employing pixelwise losses based on photo-metric or depth reconstruction errors. However, in our application we again must tackle the challenge of the asymmetry of our query (RGB) and reference (layouts) data types.…”

“…Differentiable rendering has been shown to be effective for object pose estimation [24,10]. But these works typically rely on like-for-like rendering losses, such as the pixelwise error between the rendered and target images.…”

Multi-Category Mesh Reconstruction From Image Collections

“…Recently, thanks to the development of several differentiable renderers [31,20,34,30,2], a handful of methods [17,13,16] have shown that the task can be addressed as an inverse graphics problem using fewer supervisory signals, such as 2D segmentation masks and object keypoints. Following methods have even relaxed these constraints, training without keypoint supervision [2,19,18] or known camera poses [47,7,28].…”