FabHandWear

Advanced AR/VR headsets often have a dedicated depth sensor or multiple cameras, high processing power, and a high-capacity battery to track hands or controllers. However, these approaches are not compatible with the small form factor and limited thermal capacity of lightweight AR devices. In this paper, we present RetroSphere, a self-contained 6 degree of freedom (6DoF) controller tracker that can be integrated with almost any device. RetroSphere tracks a passive controller with just 3 retroreflective spheres using a stereo pair of mass-produced infrared blob trackers, each with its own infrared LED emitters. As the sphere is completely passive, no electronics or recharging is required. Each object tracking camera provides a tiny Arduino-compatible ESP32 microcontroller with the 2D position of the spheres. A lightweight stereo depth estimation algorithm that runs on the ESP32 performs 6DoF tracking of the passive controller. Also, RetroSphere provides an auto-calibration procedure to calibrate the stereo IR tracker setup. Our work builds upon Johnny Lee's Wii remote hacks and aims to enable a community of researchers, designers, and makers to use 3D input in their projects with affordable off-the-shelf components. RetroSphere achieves a tracking accuracy of about 96.5% with errors as low as ~3.5 cm over a 100 cm tracking range, validated with ground truth 3D data obtained using a LIDAR camera while consuming around 400 mW. We provide implementation details, evaluate the accuracy of our system, and demonstrate example applications, such as mobile AR drawing, 3D measurement, etc. with our Retrosphere-enabled AR glass prototype.

show abstract

3D Deformation Capture via a Configurable Self-Sensing IMU Sensor Network

Zhou

Chen

et al. 2023

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

Motion capture technologies reconstruct human movements and have wide-ranging applications. Mainstream research on motion capture can be divided into vision-based methods and inertial measurement unit (IMU)-based methods. The vision-based methods capture complex 3D geometrical deformations with high accuracy, but they rely on expensive optical equipment and suffer from the line-of-sight occlusion problem. IMU-based methods are lightweight but hard to capture fine-grained 3D deformations. In this work, we present a configurable self-sensing IMU sensor network to bridge the gap between the vision-based and IMU-based methods. To achieve this, we propose a novel kinematic chain model based on the four-bar linkage to describe the minimum deformation process of 3D deformations. We also introduce three geometric priors, obtained from the initial shape, material properties and motion features, to assist the kinematic chain model in reconstructing deformations and overcome the data sparsity problem. Additionally, to further enhance the accuracy of deformation capture, we propose a fabrication method to customize 3D sensor networks for different objects. We introduce origami-inspired thinking to achieve the customization process, which constructs 3D sensor networks through a 3D-2D-3D digital-physical transition. The experimental results demonstrate that our method achieves comparable performance with state-of-the-art methods.

show abstract

FabHandWear

Cited by 5 publications

References 84 publications

Stepping Into the Next Decade of Ubiquitous and Pervasive Computing: UbiComp and ISWC 2021

Stepping Into the Next Decade of Ubiquitous and Pervasive Computing: UbiComp and ISWC 2021

RetroSphere

3D Deformation Capture via a Configurable Self-Sensing IMU Sensor Network

Contact Info

Product

Resources

About