Real-time neural network prediction for handling two-hands mutual occlusions

Pavllo, Dario; Delahaye, Mathias; Porssut, Thibault; Herbelin, Bruno; Boulic, Ronan

doi:10.1016/j.cagx.2019.100011

Cited by 8 publications

(6 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This tool converts markers (red LEDs) attached to the glove (Fig 1a) into 3D points in space. As optical Mocap is sensitive to visual occlusions, in particular for fingers tracking, we used the occlusion recovery process from Pavllo et al [25]. For animating the avatar, marker positions are fed to analytical Inverse Kinematic (IK) algorithms.…”

Section: Setupmentioning

confidence: 99%

“…To decide when a swap should be introduced, the system must first detect when a finger is moved (lifted) by the participant. A calibration process inspired by the work from Pavllo et al [25] was used to store all fingers' vertical's position reference when in contact with the table. Then, an offsetted hysteresis filter continuously compares fingertips' vertical positions to the reference to detect which finger is moved.…”

Section: Speed Regulationmentioning

confidence: 99%

See 1 more Smart Citation

Avatar error in your favor: Embodied avatars can fix users’ mistakes without them noticing

Delahaye

Blanke²,

Boulic³

et al. 2023

PLoS ONE

View full text Add to dashboard Cite

In immersive Virtual Reality (VR), users can experience the subjective feeling of embodiment for the avatar representing them in a virtual world. This is known to be strongly supported by a high Sense of Agency (SoA) for the movements of the avatar that follows the user. In general, users do not self-attribute actions of their avatar that are different from the one they actually performed. The situation is less clear when actions of the avatar satisfies the intention of the user despite distortions and noticeable differences between user and avatar movements. Here, a within-subject experiment was condutected to determine wether a finger swap helping users to achieve a task would be more tolerated than one penalizing them. In particular, in a context of fast-paced finger movements and with clear correct or incorrect responses, we swapped the finger animation of the avatar (e.g. user moves the index finger, the avatar moves the middle one) to either automatically correct for spontaneous mistakes or to introduce incorrect responses. Subjects playing a VR game were asked to report when they noticed the introduction of a finger swap. Results based on 3256 trials (∼24% of swaps noticed) show that swaps helping users have significantly fewer odds of being noticed (and with higher confidence) than the ones penalizing users. This demonstrates how the context and the intention for motor action are important factors for the SoA and for embodiment, opening new perspectives on how to design and study interactions in immersive VR.

show abstract

Section: Setupmentioning

confidence: 99%

Section: Speed Regulationmentioning

confidence: 99%

Avatar error in your favor: Embodied avatars can fix users’ mistakes without them noticing

Delahaye

Blanke²,

Boulic³

et al. 2023

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…This tool converts markers (red LEDs) attached to the glove (Figure 1a) into 3D points in space. As optical Mocap is sensitive to visual occlusions, in particular for fingers tracking, we used the occlusion recovery process from Pavllo et al [24]. For animating the avatar, marker positions are fed to analytical Inverse Kinematic (IK) algorithms.…”

Section: Fig 1 Experimental Contextmentioning

confidence: 99%

“…To decide when a swap should be introduced, the system must first detect when a finger is moved (lifted) by the participant. A calibration process inspired by the work from Pavllo et al [24] was used to store all fingers’ vertical’s position reference when in contact with the table. Then, an offsetted hysteresis filter continuously compares fingertips’ vertical positions to the reference to detect which finger is moved.…”

Section: Introductionmentioning

confidence: 99%

Avatar error in your favor: Embodied avatars can fix users’ mistakes without them noticing

Delahaye

Blanke

Boulic

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In immersive Virtual Reality (VR), users can experience the subjective feeling of embodiment for the avatar representing them in a virtual world. This is known to be strongly supported by a high Sense of Agency (SoA) for the movements of the avatar that follows the user. In general, users do not self-attribute actions of their avatar that are different from the one they actually performed. The situation is less clear when actions of the avatar satisfies the intention of the user despite distortions and noticeable differences between user and avatar movements. A within-subject experiment was here conducted to determine whether a finger swap helping users achieve a task would be more tolerated than one penalizing them. In particular, in a context of fast-paced finger movements and with clear correct or incorrect responses, we swapped the finger animation of the avatar (e.g. user moves the index finger, the avatar moves the middle one) to either automatically correct for spontaneous mistakes or to introduce incorrect responses. Subjects playing a VR game were asked to report when they noticed the introduction of a finger swap. Results based on 3256 trials (∼24% of swaps noticed) show that swaps helping users have significantly fewer odds of being noticed (and with higher confidence) than the ones penalizing users. This demonstrates how the context and the intention for motor action are important factors for the SoA and for embodiment, opening new perspectives on how to design and study interactions in immersive VR.

show abstract

“…Discriminative and generative are the two complementary approaches, while the hybrid approach combines both generative and discriminative approaches to complement the significant advantages of both methods (Tang et al 2018). Many of the previous works for HPE were developed based on the generative approach in which multiple hand models called hypotheses are generated and a suitable model that best matches the observed data were found (Ling et al, 2018;Pavllo et al, 2019;Xiongwei and Hoi, 2020). In the generative approach, the predicted pose parameters usually originate from the prior stage.…”

Section: Related Workmentioning

confidence: 99%

Occlusion Handler Density Networks for 3D Multimodal Joint Location of Hand Pose Hypothesis

Tito

Banzi

Mvuma

2022

TJET

View full text Add to dashboard Cite

Predicting the pose parameters during the hand pose estimation (HPE) process is an ill-posed challenge. This is due to severe self-occluded joints of the hand. The existing approaches for predicting pose parameters of the hand, utilize a single-value mapping of an input image to generate final pose output. This way makes it difficult to handle occlusion especially when it comes from the multimodal pose hypothesis. This paper introduces an effective method of handling multimodal joint occlusion using the negative log-likelihood of a multimodal mixture-of-Gaussians through a hybrid hierarchical mixture density network (HHMDN). The proposed approach generates multiple feasible hypotheses of 3D poses with visibility, unimodal and multimodal distribution units to locate joint visibility. The visible features are extracted and fed into the Convolutional Neural Networks (CNN) layer of the HHMDN for feature learning. Finally, the effectiveness of the proposed method is proved on ICVL, NYU, and BigHand public hand pose datasets. The imperative results show that the proposed method in this paper is effective as it achieves a visibility error of 30.3mm, which is less error compared to many state-of-the-art approaches that use different distributions of visible and occluded joints.

show abstract

Real-time neural network prediction for handling two-hands mutual occlusions

Cited by 8 publications

References 18 publications

Avatar error in your favor: Embodied avatars can fix users’ mistakes without them noticing

Avatar error in your favor: Embodied avatars can fix users’ mistakes without them noticing

Avatar error in your favor: Embodied avatars can fix users’ mistakes without them noticing

Occlusion Handler Density Networks for 3D Multimodal Joint Location of Hand Pose Hypothesis

Contact Info

Product

Resources

About