Nimble

Abstract

Emerging Metaverse applications demand reliable, accurate, and photorealistic reproductions of human hands to perform sophisticated operations as if in the physical world. While real human hand represents one of the most intricate coordination between bones, muscle, tendon, and skin, state-of-the-art techniques unanimously focus on modeling only the skeleton of the hand. In this paper, we present NIMBLE, a novel parametric hand model that includes the missing key components, bringing 3D hand model to a new level of realism. We first annotate muscles, bones and skins on the recent Magnetic Resonance Imaging hand (MRI-Hand) dataset [Li et al. 2021] and then register a volumetric template hand onto individual poses and subjects within the dataset. NIMBLE consists of 20 bones as triangular meshes, 7 muscle groups as tetrahedral meshes, and a skin mesh. Via iterative shape registration and parameter learning, it further produces shape blend shapes, pose blend shapes, and a joint regressor. We demonstrate applying NIMBLE to modeling, rendering, and visual inference tasks. By enforcing the inner bones and muscles to match anatomic and kinematic rules, NIMBLE can animate 3D hands to new poses at unprecedented realism. To model the appearance of skin, we further construct a photometric HandStage to acquire high-quality textures and normal maps to model wrinkles and palm print. Finally, NIMBLE also benefits learning-based hand pose and shape estimation by either synthesizing rich data or acting directly as a differentiable layer in the inference network.

Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering

“…Conventional model-based methods, such as MANO (Romero, Tzionas, and Black 2017) and Nimble (Li et al 2022b), often rely on smoothing meshes and texture maps for hand representation. Nevertheless, it typically requires costly scanning data and artistic expertise in order to achieve intricate and personalized hand meshes with texture maps.…”

“…Parametric models have been widely used in representing objects with the fixed typologies, such as the human body (Loper et al 2015;Pavlakos et al 2019;Osman, Bolkart, and Black 2020;Chen et al 2022), face (Li et al 2017;Hong et al 2022), hands (Romero, Tzionas, and Black 2017;Li et al 2022b), and animals (Zuffi et al 2017). These models enable the transformation of mesh geometry by adjusting model parameters corresponding to pose and shape variations.…”

“…Moreover, hand pose can be effectively estimated by images (Li, Gao, and Sang 2021) or point clouds (Cheng et al 2022;. In hand reconstruction, the parametric models like MANO (Romero, Tzionas, and Black 2017), NIMBLE (Li et al 2022b), have been used to recover the hand in the input image (Boukhayma, de Bem, and Torr 2019;Hasson et al 2019;Kong et al 2022;Cao et al 2021;Doosti et al 2020;Hasson et al 2020). In (Fan et al 2021;Ren, Zhu, and Zhang 2023;Kim, Kim, and Baek 2021;Zhang et al 2021;Li et al 2022a), the parametric models are employed to reconstruct two hands, where hand interactions and gestures could be simulated.…”

A survey on generative 3D digital humans based on neural networks: representation, rendering, and learning

“…为此, 一些 近期工作结合人体解剖学理论, 在手部 [217∼219] 、骨骼肌肉 [220∼222] 等部位构建更符合真人运动规律 的数字人模型. 如 NIMBLE [217] 针对手部构建模型, 使用核磁共振数据构建了骨骼、肌肉和皮肤的生 物模型, 并在光场下采集真实手部纹理作为贴图, 实现了虚拟手的生成. Generative GaitNet [220] 构建 了真人全身肌肉和骨骼模型, 并基于物理约束实现了实时的人体步态模拟.…”

Bidirectional GaitNet: A Bidirectional Prediction Model of Human Gait and Anatomical Conditions

“…Anthropometric scaling of a musculoskeletal model can generate a range of models with different heights, weights, and limb lengths [Ryu et al 2021]. Building accurate individualized models often requires expensive medical images (e.g., CT and MRI) and labor-intensive image labeling [Levin et al 2011;Li et al 2022;Matias et al 2009].…”