HybridTrak: Adding Full-Body Tracking to VR Using an Off-the-Shelf Webcam

Yang, Junrui; Chen, Tuochao; Qin, Fang; Lam, Monica S.; Landay, James A.

doi:10.1145/3491102.3502045

Cited by 14 publications

(2 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ž In addition, it has been argued for many years that HCI research should reposition itself away from traditional modes of interaction that focus on verbal, text-based, keyboard and mouse driven conversational interaction to embodied interaction where the context of human physicality and physical objects supplement cognitive approaches [5]. Virtual reality systems are designed to support multimodal activities such as head and hand movements [6,7] and video capture systems process full body motion [8ś10]. While these and other prototypes continue to be evaluated for how they incorporate gesture and head movements [11,12], support for full body interaction remains elusive.…”

Section: Literature Reviewmentioning

confidence: 99%

Stage, Studio and Screen: Reimagining Dance Online

Sabherwal

Eden

2023

Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Dance is a field of study expressed through a series of improvised or choreographed movements and steps that involve arms, legs, and torsos extending across physical space. When these activities shift online, the challenges for performance and teaching are more novel than those found in verbal or text-based communication. This paper discusses preliminary results from a qualitative study conducted to understand how performers and students adapted to online spaces when in-person events were not available. We examine how performers reconstructed the stage and overcame obstacles to building rapport with audiences. We also investigate how students assembled makeshift dance studios in the home and challenges they faced when trying to make sense of choreographed instructions. Preliminary analysis shows that existing technologies lack support for performing and learning dance online, and we conclude with suggestions for how more sophisticated systems might be designed to support embodied knowledge production and transfer.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Stage, Studio and Screen: Reimagining Dance Online

Sabherwal

Eden

2023

Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…Various user input data from consumer‐grade devices can be used to synthesize full‐body animation for characters in real time. For example, some studies used optical data from egocentric cameras mounted in a baseball cap [XCZ*19], a HMD [TAP*20; YCQ*22], controllers [ASF*22], or glasses [ZWMF21] to estimate the body pose. Egocentric cameras suffer from extreme perspective distortion and self‐occlusion that lead to inadequate tracking information for the lower body.…”

Section: Related Workmentioning

confidence: 99%

Combining Motion Matching and Orientation Prediction to Animate Avatars for Consumer‐Grade VR Devices

Luis

Yun

Andújar

et al. 2022

Computer Graphics Forum

View full text Add to dashboard Cite

The animation of user avatars plays a crucial role in conveying their pose, gestures, and relative distances to virtual objects or other users. Self‐avatar animation in immersive VR helps improve the user experience and provides a Sense of Embodiment. However, consumer‐grade VR devices typically include at most three trackers, one at the Head Mounted Display (HMD), and two at the handheld VR controllers. Since the problem of reconstructing the user pose from such sparse data is ill‐defined, especially for the lower body, the approach adopted by most VR games consists of assuming the body orientation matches that of the HMD, and applying animation blending and time‐warping from a reduced set of animations. Unfortunately, this approach produces noticeable mismatches between user and avatar movements. In this work we present a new approach to animate user avatars that is suitable for current mainstream VR devices. First, we use a neural network to estimate the user's body orientation based on the tracking information from the HMD and the hand controllers. Then we use this orientation together with the velocity and rotation of the HMD to build a feature vector that feeds a Motion Matching algorithm. We built a MoCap database with animations of VR users wearing a HMD and used it to test our approach on both self‐avatars and other users' avatars. Our results show that our system can provide a large variety of lower body animations while correctly matching the user orientation, which in turn allows us to represent not only forward movements but also stepping in any direction.

show abstract