Real-Time Marker-Based Finger Tracking with Neural Networks

Pavllo, Dario; Porssut, Thibault; Herbelin, Bruno; Boulic, Ronan

doi:10.1109/vr.2018.8446173

Cited by 7 publications

(10 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data-driven methods learn from a large database to acquire intrinsic knowledge of MoCap data, such as KD-tree Tautges et al 2011], local PCA [Chai and Hodgins 2005;Liu and McMillan 2006], self-similarity [Aristidou et al 2018], sparse encoding [Wang et al 2016;Xiao et al 2015], and model averaging [Tits et al 2018]. With the advancement of deep-learning, a number of neural-based methods have emerged [Chen et al 2021;Ghorbani and Black 2021;Holden 2018;Pavllo et al 2018;Perepichka et al 2019]. SOMA [Ghorbani and Black 2021] uses a transformerbased network to automatically label the marker point cloud.…”

Section: Related Workmentioning

confidence: 99%

“…The majority of works focus on repairing occluded markers and solving motions. [Pavllo et al 2018] uses auto-encoder-based models to recover occluded hand markers and solve hand motions. [Perepichka et al 2019] presents a missing marker completion method by comparing motions generated by commercial software with those generated by a neural solver, the accuracy of these methods is determined by the neural solver.…”

Section: Related Workmentioning

confidence: 99%

“…simultaneous occlusions of multiple markers, and varying duration of occlusions. Recently, data-driven methods [Chen et al 2021;Ghorbani and Black 2021;Holden 2018;Pavllo et al 2018;Perepichka et al 2019;Tits et al 2018;Wang et al 2016;Xiao et al 2015] have been employed to address the aforementioned limitations. For instance, MoCap-Solver [Chen et al 2021] solves motions and reconstructs clean markers by separately encoding motion, marker configuration, and skeleton using different neural networks, and achieves the state-of-the-art results.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Locality-based Neural Solver for Optical Motion Capture

Pan,

Zheng,

Jiang

et al. 2023

SIGGRAPH Asia 2023 Conference Papers

View full text Add to dashboard Cite

We present a novel locality-based learning method for cleaning and solving optical motion capture data. Given noisy marker data, we propose a new heterogeneous graph neural network which treats markers and joints as different types of nodes, and uses graph convolution operations to extract the local features of markers and joints and transform them to clean motions. To deal with anomaly markers (e.g. occluded or with big tracking errors), the key insight is that a marker's motion shows strong correlations with the motions of its immediate neighboring markers but less so with other markers, a.k.a. locality, which enables us to efficiently fill

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Locality-based Neural Solver for Optical Motion Capture

Pan,

Zheng,

Jiang

et al. 2023

SIGGRAPH Asia 2023 Conference Papers

View full text Add to dashboard Cite

show abstract

“…Qian et al [36] leverage the network in [4] to obtain the motion parameters of MANO and further to refine the mesh model by photometric loss. Other deep learning-based approaches either leverage depth information [29,32] or combine image and depth information together. As pose data usually distributes near the mean pose of the dataset, neural networks are prone to smooth the pose and make the result near the mean [41].…”

Section: Hand Pose Reconstructionmentioning

confidence: 99%

3D hand reconstruction from a single image based on biomechanical constraints

Liu

et al. 2021

Vis Comput

View full text Add to dashboard Cite

This paper investigates the estimate of motion parameters from 3D hand joint positions. We formulate the issue as an inverse kinematics problem with biomechanical constraints and propose a fast and robust iterative approach to address the constrained optimization. It elaborately designs a coordinate descent algorithm to decompose the problem into a sequence of decisions on the transformation around each kinematic node (i.e., joint), while the decision for each node is equivalent to a point matching problem. Addressing the whole optimization then amounts to considering all nodes of the kinematic tree from its root to leaves one by one. This not only accelerates the process but also improves the accuracy of the solution of the inverse kinematic optimization. Experiments show that our approach is able to yield results comparable to and even better than those by the state-of-the-art methods.

show abstract

“…Zhou et al [136] proposed a deep model that confirms the geometric validity of pose prediction by using a forward kinematics-based layer, which fully exploits prior knowledge in a generative model for hand geometry estimation. Pavllo et al [137] proposed a real-time neural network that uses a motion capture system containing cameras and active markers to track hands and fingers. It copes well with occlusion and completely reconstructs hand posture.…”

Section: Hand Gesture Recognitionmentioning

confidence: 99%

VR content creation and exploration with deep learning: A survey

Wang

Lyu

et al. 2020

Comp. Visual Media

View full text Add to dashboard Cite

Virtual reality (VR) offers an artificial, computer generated simulation of a real life environment. It originated in the 1960s and has evolved to provide increasing immersion, interactivity, imagination, and intelligence. Because deep learning systems are able to represent and compose information at various levels in a deep hierarchical fashion, they can build very powerful models which leverage large quantities of visual media data. Intelligence of VR methods and applications has been significantly boosted by the recent developments in deep learning techniques. VR content creation and exploration relates to image and video analysis, synthesis and editing, so deep learning methods such as fully convolutional networks and general adversarial networks are widely employed, designed specifically to handle panoramic images and video and virtual 3D scenes. This article surveys recent research that uses such deep learning methods for VR content creation and exploration. It considers the problems involved, and discusses possible future directions in this active and emerging research area. Keywords virtual reality; deep learning; neural networks; 360 • image and video virtual content

show abstract

Real-Time Marker-Based Finger Tracking with Neural Networks

Cited by 7 publications

References 8 publications

A Locality-based Neural Solver for Optical Motion Capture

A Locality-based Neural Solver for Optical Motion Capture

3D hand reconstruction from a single image based on biomechanical constraints

VR content creation and exploration with deep learning: A survey

Contact Info

Product

Resources

About