Neural 3D Mesh Renderer

Kato, Hiroharu; Ushiku, Yoshitaka; Harada, Tatsuya

doi:10.1109/cvpr.2018.00411

Cited by 934 publications

(902 citation statements)

References 17 publications

Supporting

Mentioning

859

Contrasting

Unclassified

Order By: Relevance

“…Human body surface is represented as a 3D triangular mesh, and body posing is obtained by standard linear blend skinning. To fully compare a reconstructed 3D human body to a 2D observation of it, we integrate a differentiable renderer, i.e., a computer graphics technique that creates a 2D image from a 3D object using differentiable operations [31,23], and develop an end-to-end weakly-supervised training scheme.…”

Section: Dense Render-and-comparementioning

confidence: 99%

“…In [31], the authors approximate derivatives at occlusion boundaries which are discontinuous, while colors are interpolated between vertices (i.e., there is no differentiation with respect to texture). In [23], the authors obtain approximate gradients by blurring image to avoid sudden pixel color change. This produces non-zero gradients and enables gradient-flow between pixel (color) value to vertex position.…”

Section: Dense Render-and-comparementioning

confidence: 99%

See 1 more Smart Citation

DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare

Zhu

Tung

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

183

112

View full text Add to dashboard Cite

We present DenseRaC, a novel end-to-end framework for jointly estimating 3D human pose and body shape from a monocular RGB image. Our two-step framework takes the body pixel-to-surface correspondence map (i.e., IUV map) as proxy representation and then performs estimation of parameterized human pose and shape. Specifically, given an estimated IUV map, we develop a deep neural network optimizing 3D body reconstruction losses and further integrating a render-and-compare scheme to minimize differences between the input and the rendered output, i.e., dense body landmarks, body part masks, and adversarial priors. To boost learning, we further construct a large-scale synthetic dataset (MOCA) utilizing web-crawled Mocap sequences, 3D scans and animations. The generated data covers diversified camera views, human actions and body shapes, and is paired with full ground truth. Our model jointly learns to represent the 3D human body from hybrid datasets, mitigating the problem of unpaired training data. Our experiments show that DenseRaC obtains superior performance against state of the art on public benchmarks of various humanrelated tasks.

show abstract

Section: Dense Render-and-comparementioning

confidence: 99%

Section: Dense Render-and-comparementioning

confidence: 99%

DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare

Zhu

Tung

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

183

112

View full text Add to dashboard Cite

show abstract

“…In the case of occlusions, i.e., when multiple triangles overlap a single pixel, we consider the triangle with the largest motion to be in front. This is a commonly used heuristics [13] which often holds true in practice (in particular when image motion is dominated by camera motion). Note that due to the linear interpolation, the warping function W 0→i is piecewise smooth.…”

Section: B Reblurringmentioning

confidence: 99%

Self-Supervised Linear Motion Deblurring

Liu

Janai

Pollefeys

et al. 2020

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Motion blurry images challenge many computer vision algorithms, e.g., feature detection, motion estimation, or object recognition. Deep convolutional neural networks are stateof-the-art for image deblurring. However, obtaining training data with corresponding sharp and blurry image pairs can be difficult. In this paper, we present a differentiable reblur model for selfsupervised motion deblurring, which enables the network to learn from real-world blurry image sequences without relying on sharp images for supervision. Our key insight is that motion cues obtained from consecutive images yield sufficient information to inform the deblurring task. We therefore formulate deblurring as an inverse rendering problem, taking into account the physical image formation process: we first predict two deblurred images from which we estimate the corresponding optical flow. Using these predictions, we re-render the blurred images and minimize the difference with respect to the original blurry inputs. We use both synthetic and real dataset for experimental evaluations. Our experiments demonstrate that self-supervised single image deblurring is really feasible and leads to visually compelling results. Both the code and datasets are available at https://github.com/ethliup/SelfDeblur.

show abstract

“…Loper and Black (2014) is one of the well known methods for differential rendering, but has limited applicability due to high computational and memory costs. Kato et al (2017); Rezende et al (2016) approximate the gradients of the rendering process and are often limited to a rasterization based rendering scheme. OpenDR (Loper and Black 2014), as used by Henderson and Ferrari (2018), applies first order Taylor approximation to compute gradients.…”

Section: Differentiable Renderingmentioning

confidence: 99%

Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation

Rajeswar

Mannan²,

Golemo

et al. 2020

Int J Comput Vis

View full text Add to dashboard Cite

We infer and generate three-dimensional (3D) scene information from a single input image and without supervision. This problem is under-explored, with most prior work relying on supervision from, e.g., 3D groundtruth, multiple images of a scene, image silhouettes or key-points. We propose Pix2Shape, an approach to solve this problem with four component: (i) an encoder that infers the latent 3D representation from an image, (ii) a decoder that generates an explicit 2.5D surfelbased reconstruction of a scene -from the latent code -(iii) a differentiable renderer that synthesizes a 2D image from the surfel representation, and (iv) a critic network trained to discriminate between images generated by the decoder-renderer and those from a training distribution. Pix2Shape can generate complex 3D scenes that scale with the view-dependent on-screen resolution, unlike representations that capture world-space resolution, i.e., voxels or meshes. We show that Pix2Shape learns a consistent scene representation in its encoded latent space, and that the decoder can then be applied to this latent representation in order to synthesize the scene from a novel viewpoint. We evaluate Pix2Shape with experiments on the ShapeNet dataset as well as on a novel benchmark we developed -called 3D-IQTTto evaluate models based on their ability to enable 3d spatial reasoning. Qualitative and quantitative evaluation demonstrate Pix2Shape's ability to solve scene reconstruction, generation and understanding tasks.

show abstract

Neural 3D Mesh Renderer

Cited by 934 publications

References 17 publications

DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare

DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare

Self-Supervised Linear Motion Deblurring

Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation

Contact Info

Product

Resources

About