MoCap-solver

Abstract

In a conventional optical motion capture (MoCap) workflow, two processes are needed to turn captured raw marker sequences into correct skeletal animation sequences. Firstly, various tracking errors present in the markers must be fixed ( cleaning or refining ). Secondly, an agent skeletal mesh must be prepared for the actor/actress, and used to determine skeleton information from the markers ( re-targeting or solving ). The whole process, normally referred to as solving MoCap data, is extremely time-consuming, labor-intensive, and usually the most costly part of animation production. Hence, there is a great demand for automated tools in industry. In this work, we present MoCap-Solver, a production-ready neural solver for optical MoCap data. It can directly produce skeleton sequences and clean marker sequences from raw MoCap markers, without any tedious manual operations. To achieve this goal, our key idea is to make use of neural encoders concerning three key intrinsic components: the template skeleton, marker configuration and motion, and to learn to predict these latent vectors from imperfect marker sequences containing noise and errors. By decoding these components from latent vectors, sequences of clean markers and skeletons can be directly recovered. Moreover, we also provide a novel normalization strategy based on learning a pose-dependent marker reliability function, which greatly improves system robustness. Experimental results demonstrate that our algorithm consistently outperforms the state-of-the-art on both synthetic and real-world datasets.

A Locality-based Neural Solver for Optical Motion Capture

“…Data-driven methods learn from a large database to acquire intrinsic knowledge of MoCap data, such as KD-tree Tautges et al 2011], local PCA [Chai and Hodgins 2005;Liu and McMillan 2006], self-similarity [Aristidou et al 2018], sparse encoding [Wang et al 2016;Xiao et al 2015], and model averaging [Tits et al 2018]. With the advancement of deep-learning, a number of neural-based methods have emerged [Chen et al 2021;Ghorbani and Black 2021;Holden 2018;Pavllo et al 2018;Perepichka et al 2019]. SOMA [Ghorbani and Black 2021] uses a transformerbased network to automatically label the marker point cloud.…”

“…Their method solves the motion and skeleton frame by frame and necessitates smoothing as a post-processing step for temporal motion continuity. MoCap-Solver [Chen et al 2021] solves motions in a temporal window by encoding body shapes, marker distributions, and motions separately to ensure temporal continuity. Their method reconstructs clean markers based on skinning functions with solved motions, which ignores the complexity of marker motions (e.g.…”

“…In contrast, our method fills the occlusion by incorporating hand-crafted priors and learned intrinsic priors, which accurately reconstruct occluded markers. Furthermore, previous methods [Chen et al 2021;Holden 2018] do not explicitly quantify the local feature of markers, which negatively affects solving result of certain large body motions and fine multi-fingered hand motions. Unlike them, our method extracts local features by constructing a heterogeneous graph that distinguishes markers and joints as different types of nodes and performing graph convolution operations on it, which significantly improves solving accuracy.…”

“…Overall, we observe that the state-of-the-art data-driven methods [Chen et al 2021;Holden 2018] have three main limitations. First, they solve for motion and reconstruct clean markers using skinning functions, which ignore the complexity of marker motions (e.g., sliding across the body surface) and may introduce additional errors due to motion solving errors.…”

“…Recently, data-driven methods [Chen et al 2021;Ghorbani and Black 2021;Holden 2018;Pavllo et al 2018;Perepichka et al 2019;Tits et al 2018;Wang et al 2016;Xiao et al 2015] have been employed to address the aforementioned limitations. For instance, MoCap-Solver [Chen et al 2021] solves motions and reconstructs clean markers by separately encoding motion, marker configuration, and skeleton using different neural networks, and achieves the state-of-the-art results. However, dealing with noisy marker data with outliers and large simultaneous occlusions remains difficult.…”