Off-The-Shelf Stylus: Using XR Devices for Handwriting and Sketching on Physically Aligned Virtual Surfaces

Abstract

This article introduces the Off-The-Shelf Stylus (OTSS), a framework for 2D interaction (in 3D) as well as for handwriting and sketching with digital pen, ink, and paper on physically aligned virtual surfaces in Virtual, Augmented, and Mixed Reality (VR, AR, MR: XR for short). OTSS supports self-made XR styluses based on consumer-grade six-degrees-of-freedom XR controllers and commercially available styluses. The framework provides separate modules for three basic but vital features: 1) The stylus module provides stylus construction and calibration features. 2) The surface module provides surface calibration and visual feedback features for virtual-physical 2D surface alignment using our so-called 3ViSuAl procedure, and surface interaction features. 3) The evaluation suite provides a comprehensive test bed combining technical measurements for precision, accuracy, and latency with extensive usability evaluations including handwriting and sketching tasks based on established visuomotor, graphomotor, and handwriting research. The framework’s development is accompanied by an extensive open source reference implementation targeting the Unity game engine using an Oculus Rift S headset and Oculus Touch controllers. The development compares three low-cost and low-tech options to equip controllers with a tip and includes a web browser-based surface providing support for interacting, handwriting, and sketching. The evaluation of the reference implementation based on the OTSS framework identified an average stylus precision of 0.98 mm (SD = 0.54 mm) and an average surface accuracy of 0.60 mm (SD = 0.32 mm) in a seated VR environment. The time for displaying the stylus movement as digital ink on the web browser surface in VR was 79.40 ms on average (SD = 23.26 ms), including the physical controller’s motion-to-photon latency visualized by its virtual representation (M = 42.57 ms, SD = 15.70 ms). The usability evaluation (N = 10) revealed a low task load, high usability, and high user experience. Participants successfully reproduced given shapes and created legible handwriting, indicating that the OTSS and it’s reference implementation is ready for everyday use. We provide source code access to our implementation, including stylus and surface calibration and surface interaction features, making it easy to reuse, extend, adapt and/or replicate previous results (https://go.uniwue.de/hci-otss).

Text Input for Non-Stationary XR Workspaces: Investigating Tap and Word-Gesture Keyboards in Virtual and Augmented Reality

“…Both controllers and hand tracking enable direct interactions with virtual objects. For consumer XR devices, the controller enables more accurate motion detection than integrated hand tracking [2,12,39,40,84]. In addition, the controllers' internal sensors enable higher tolerance to occlusions and faster response to motion.…”

“…Professional systems such as OptiTrack Motion Capture provide better accuracy and faster motion detection for hand tracking [20,30,81], but they are mostly not available in mobile XR workspaces with consumer XR hardware. Kern et al [40] showed that most XR controllers can also be used as XR styluses. While the typical controller grip (power grip) requires wrist and arm movements, holding the controller upside down in a pen-like posture (precision grip) enables fine-grained movements that are essential for accurate drawing, handwriting, and direct interactions with small keys on a virtual keyboard, resulting in fewer errors and higher performance [5,73].…”

Breaking Plausibility Without Breaking Presence - Evidence For The Multi-Layer Nature Of Plausibility

“…Prominent XR use-cases now encompass education and training, health and rehabilitation, collaboration and social VR, entertainment and recreation, and many more. For example, XR applications help people overcome bad habituates [26], recover from surgery [8], learn remotely [15], exchange interculturally [10], or even understand Artificial Intelligence [27]. Overall, the XR use-cases comprise some very different goals as well as partly unique requirements.…”

Extended Reality for Knowledge Work in Everyday Environments

“…A virtual representation of the users hand and the tablet is presented on the large display. Kern et al [52] introduced a framework for 2D interaction in 3D including digital pen and paper on physically aligned surfaces and a study (n=10) showed that the technique resulted in low task load and high usability.…”

“…As has been seen, many proposed techniques include a touch surface [68,10,36,57], which enables fine grained sensing of writing and touch, while supporting the users fingers or stylus, and has found to perform better than mid-air techniques [68,36,81,52]. The large display space of XR enable display of more data than typical physical displays, yet the need to manipulate data over such large space efficiently without generating fatigue of the user is a challenge.…”