Driving-signal aware full-body avatars

Abstract

We present a learning-based method for building driving-signal aware full-body avatars. Our model is a conditional variational autoencoder that can be animated with incomplete driving signals, such as human pose and facial keypoints, and produces a high-quality representation of human geometry and view-dependent appearance. The core intuition behind our method is that better drivability and generalization can be achieved by disentangling the driving signals and remaining generative factors, which are not available during animation. To this end, we explicitly account for information deficiency in the driving signal by introducing a latent space that exclusively captures the remaining information, thus enabling the imputation of the missing factors required during full-body animation, while remaining faithful to the driving signal. We also propose a learnable localized compression for the driving signal which promotes better generalization, and helps minimize the influence of global chance-correlations often found in real datasets. For a given driving signal, the resulting variational model produces a compact space of uncertainty for missing factors that allows for an imputation strategy best suited to a particular application. We demonstrate the efficacy of our approach on the challenging problem of full-body animation for virtual telepresence with driving signals acquired from minimal sensors placed in the environment and mounted on a VR-headset.

RelightableHands: Efficient Neural Relighting of Articulated Hand Models

“…As demonstrated in [47], the articulated MVP improves fidelity over mesh-based representations due to its volumetric nature while being computationally efficient for real-time rendering. Additionally, it only requires a coarse mesh from an LBS model as guidance, in contrast to prior mesh-based works [2,29], which rely on precise surface tracking.…”

“…Additionally, joint features are input at the lowest resolution level of the U-Net layer such that the resulting appearance explicitly accounts for pose-dependent texture changes, such as small wrinkles, that may not be represented by the primitive geometry. We use a spatially aligned joint feature encoder J t (θ)∈R 64×64×64 in UV space as in [2]. Our loss is expressed by…”

MoRF: Mobile Realistic Fullbody Avatars from a Monocular Video

“…Modeling geometry of dynamic non-rigid scenes is considered in a number of recent approaches either for capturing human actors [5,54,29,27,30,35,16,20,17,59] or general scenes [32,45,41]. Towards this end, 3D scans are required for supervision in [20,17,35,54,29] to learn rigged human geometry.…”

“…Towards this end, 3D scans are required for supervision in [20,17,35,54,29] to learn rigged human geometry. Likewise, [32,30,16,5,48,27] utilize multi-view data to capture appearance and produce photo-realistic avatars under arbitrary viewpoints and in arbitrary poses. Several methods [59,45,41] use monocular videos but allow free-viewpoint rendering only.…”

Optimization of Turbulence Model Parameters Using the Global Search Method Combined with Machine Learning

“…A learning-based method for building driving-signal aware full-body avatars was presented in [27]. The model was a conditional variational auto-encoder that could be animated with incomplete driving signals, such as human pose and facial keypoints, and produced a high-quality representation of human geometry and view-dependent appearance.…”