DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Cameras

Zheng, Yang; Shao, Ruizhi; Zhang, Yuxiang; Yu, Tao; Zheng, Zerong; Dai, Qionghai; Liu, Yebin

doi:10.48550/arxiv.2105.00261

Cited by 20 publications

(36 citation statements)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The problem of digital reconstruction of humans is a long-standing problem in computer vision and computer graphics. Traditional methods usually achieve high quality with complicated capture setups such as multi-view capture studio [8,36,35] or RGB-D camera arrays [33,30]. To reduce capture efforts, recent methods leverage deep neural networks to directly reconstruct 3d humans from even single images [20,27,14,7].…”

Section: Related Workmentioning

confidence: 99%

Neural Capture of Animatable 3D Human from Monocular Video

Te¹,

Li²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present a novel paradigm of building an animatable 3D human representation from a monocular video input, such that it can be rendered in any unseen poses and views. Our method is based on a dynamic Neural Radiance Field (NeRF) rigged by a mesh-based parametric 3D human model serving as a geometry proxy. Previous methods usually rely on multi-view videos or accurate 3D geometry information as additional inputs; besides, most methods suffer from degraded quality when generalized to unseen poses. We identify that the key to generalization is a good input embedding for querying dynamic NeRF: A good input embedding should define an injective mapping in the full volumetric space, guided by surface mesh deformation under pose variation. Based on this observation, we propose to embed the input query with its relationship to local surface regions spanned by a set of geodesic nearest neighbors on mesh vertices. By including both position and relative distance information, our embedding defines a distance-preserved deformation mapping and generalizes well to unseen poses. To reduce the dependency on additional inputs, we first initialize per-frame 3D meshes using off-the-shelf tools and then propose a pipeline to jointly optimize NeRF and refine the initial mesh. Extensive experiments show our method can synthesize plausible human rendering results under unseen poses and views.

show abstract

Section: Related Workmentioning

confidence: 99%

Neural Capture of Animatable 3D Human from Monocular Video

Te¹,

Li²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For multi-view human reconstruction, PIFu uses a naive average pooling operation for multi-view feature fusion, which is not efficient enough to fuse the geometry details that observed in the multi-view input images. [18] uses SMPL as the geometry prior to solve the occlusion problem in multi-person reconstruction, but the usage of SMPL not only makes the reconstruction results might be pixel-misaligned, but also makes it unable to reconstruct the human cases with loose clothes, such as long dress. In contrast, without using any geometry prior, we present a pixel-aligned spatial transformer for multi-view feature fusion, enabling us to produce pixel-aligned highly detailed reconstruction results.…”

Section: Related Workmentioning

confidence: 99%

“…3. Though the usage of SMPL model makes the reconstruction results of [18] more robust, it will cause two problems: 1) the reconstruction results might be pixelmisaligned as the SMPL models do not contain any geometry details, this misalignment will result in artifacts in the rendering results, 2) it will be difficult to reconstruct humans with loose clothes, such as long dress. For this end, we have to find a way to robustly reconstruct humans without using any geometry prior, enabling us to produce pixel-aligned reconstruction results and handle the humans cases with loose clothes.…”

Section: Preliminarymentioning

confidence: 99%

“…This average operation regard each view equally, but in fact, for each 3D point in world space, different views should have different weights for the occupancy estimation of the 3D point. [18] uses human shape and pose prior SMPL [6] for multi-person reconstruction using sparse views. The usage of SMPL enables it to solve the occlusion problems caused by the interaction of multi-human.…”

Section: Preliminarymentioning

confidence: 99%

See 1 more Smart Citation

HDhuman: High-quality Human Performance Capture with Sparse Views

Zhou¹,

Chen²,

Shao³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we introduce HDhuman, a method that addresses the challenge of novel view rendering of human performers that wear clothes with complex texture patterns using a sparse set of camera views. Although some recent works have achieved remarkable rendering quality on humans with relatively uniform textures using sparse views, the rendering quality remains limited when dealing with complex texture patterns as they are unable to recover the high-frequency geometry details that observed in the input views. To this end, the proposed HDhuman uses a human reconstruction network with a pixel-aligned spatial transformer and a rendering network that uses geometry-guided pixel-wise feature integration to achieve high-quality human reconstruction and rendering. The designed pixel-aligned spatial transformer calculates the correlations between the input views, producing human reconstruction results with high-frequency details. Based on the surface reconstruction results, the geometry-guided pixel-wise visibility reasoning provides guidance for multi-view feature integration, enabling the rendering network to render high-quality images at 2k resolution on novel views. Unlike previous neural rendering works that always need to train or fine-tune an independent network for a different scene, our method is a general framework that is able to generalize to novel subjects. Experiments show that our approach outperforms all the prior generic or specific methods on both synthetic data and real-world data.

show abstract

“…However, these are expensive commercial datasets and therefore their accessibility is limited to companies that have the financial resources to purchase them . Consequently, researchers have created other databases of 3D human models to train deep learning methods [Zheng et al, 2019 Irish Machine Vision and Image Processing Conference (IMVIP) | DOI: 10.56541/SOWW6683 ISBN 978-0-9934207-7-1 , Pumarola et al, 2019, Gabeur et al, 2019, Caliskan et al, 2020. But these datasets are limited in the quality of the models and the number poses because of the effort and equipment needed for the capture and preparation of the data.…”

Section: Introductionmentioning

confidence: 99%

Geometrically reconstructing confocal microscopy images for modelling the retinal microvasculature as a 3D cylindrical network

Troendle¹,

Barabás²,

Curtis³

2022

24th Irish Machine Vision and Image Processing Conference

View full text Add to dashboard Cite

Microvascular networks can be modelled as a network of connected cylinders. Presently, however, there are limited approaches with which to recover these networks from biomedical images. We have therefore developed and implemented computer algorithms to geometrically reconstruct three-dimensional (3D) retinal microvascular networks from micrometre-scale imagery, resulting in a concise representation of two endpoints and radius for each cylinder detected within a delimited text file. This format is suitable for a variety of purposes, including efficient simulations of molecular delivery. Here, we detail a semi-automated pipeline consisting of the detection of retinal microvascular volumes within 3D imaging datasets, the enhancement and analysis of these volumes for reconstruction, and the geometric construction algorithm itself, which converts voxel data into representative 3D cylindrical objects.

show abstract

DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Cameras

Cited by 20 publications

References 70 publications

Neural Capture of Animatable 3D Human from Monocular Video

Neural Capture of Animatable 3D Human from Monocular Video

HDhuman: High-quality Human Performance Capture with Sparse Views

Geometrically reconstructing confocal microscopy images for modelling the retinal microvasculature as a 3D cylindrical network

Contact Info

Product

Resources

About