VRSketchPen: Unconstrained Haptic Assistance for Sketching in Virtual 3D Environments

Elsayed, Hesham; Machuca, Mayra Donaji Barrera; Schaarschmidt, Christian; Marky, Karola; Müller, Florian; Riemann, Jan; Matviienko, Andrii; Schmitz, Martin; Weigel, Martin; Mühlhäuser, Max

doi:10.1145/3385956.3418953

Cited by 33 publications

(13 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, dedicated XR stylus devices have been announced, e.g., Logitech's VR Ink 1 , Holo-Light's Stylus XR 2 , or Wacom's VR Pen 3 . Also, the HCI community developed several input devices like the Flashpen (Romat et al, 2021), the OVR Stylus (Jackson, 2020), the DodecaPen (Wu et al, 2017), the Elastylus (Lyu et al, 2015), the SenStylus (Fiorentino and Uva, 2005), the VRSketchPen (Elsayed et al, 2020), and other not formally named devices (Viciana-Abad et al, 2010;Arora et al, 2017;Wacker et al, 2018;Chen et al, 2019;Pham and Stuerzlinger, 2019;Drey et al, 2020;Gesslein et al, 2020;Li et al, 2020;Bowers et al, 2021). XR styluses typically require additional tracking sensors, like a general-use motion capture system, finger tracking, custommade tracking systems, or haptic devices.…”

Section: Virtual Reality Stylusesmentioning

confidence: 99%

“…In addition, researchers combined their tracking solutions with 2D tracking devices like graphics tablets to provide a solid surface for precise and accurate handwriting and sketching (Billinghurst et al, 1997;Poupyrev et al, 1998;Chen et al, 2019;Wang et al, 2019;Gesslein et al, 2020;Hsu et al, 2021). While self-made solutions can provide high tracking quality depending on the deployed tracking system, button interactions, and even haptic feedback (Bowers et al, 2020;Drey et al, 2020;Elsayed et al, 2020;Jackson, 2020;Bowers et al, 2021), they frequently require 3D printed components (Bowers et al, 2020(Bowers et al, , 2021Jackson, 2020) or complex constructions (Fiorentino and Uva, 2005), which limits the availability of these devices. Also, the necessity of microcontrollers (Fiorentino and Uva, 2005;Bowers et al, 2020;Jackson, 2020;Romat et al, 2021), which require programming skills, may seem simple for technically experienced users, but makes these devices unattainable and unusable for beginners and inexperienced users.…”

Section: Virtual Reality Stylusesmentioning

confidence: 99%

“…Dedicated pen-like input devices (so-called styluses) have recently been announced as pilot versions. While these devices are expensive or unavailable, they, like many scientific stylus prototypes, usually require external hardware components such as additional six-degrees-of-freedom (6DoF) tracking systems, microcontrollers, or 3D printed parts, and often include custommade input metaphors and interface implementations (Poupyrev et al, 1998;Fiorentino and Uva, 2005;Arora et al, 2017;Wu et al, 2017;Chen et al, 2019;Elsayed et al, 2020;Jackson, 2020;Romat et al, 2021). Other approaches try to overcome limited availability, high pricing, or a complex construction process by basing their XR styluses on consumer-grade XR devices (e.g., HTC Vive Controller or Oculus Touch controller) held upside down (Pham and Stuerzlinger, 2019;Wang et al, 2019;Bowers et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Off-The-Shelf Stylus: Using XR Devices for Handwriting and Sketching on Physically Aligned Virtual Surfaces

Kern

Kullmann

Ganal

et al. 2021

Front. Virtual Real.

View full text Add to dashboard Cite

This article introduces the Off-The-Shelf Stylus (OTSS), a framework for 2D interaction (in 3D) as well as for handwriting and sketching with digital pen, ink, and paper on physically aligned virtual surfaces in Virtual, Augmented, and Mixed Reality (VR, AR, MR: XR for short). OTSS supports self-made XR styluses based on consumer-grade six-degrees-of-freedom XR controllers and commercially available styluses. The framework provides separate modules for three basic but vital features: 1) The stylus module provides stylus construction and calibration features. 2) The surface module provides surface calibration and visual feedback features for virtual-physical 2D surface alignment using our so-called 3ViSuAl procedure, and surface interaction features. 3) The evaluation suite provides a comprehensive test bed combining technical measurements for precision, accuracy, and latency with extensive usability evaluations including handwriting and sketching tasks based on established visuomotor, graphomotor, and handwriting research. The framework’s development is accompanied by an extensive open source reference implementation targeting the Unity game engine using an Oculus Rift S headset and Oculus Touch controllers. The development compares three low-cost and low-tech options to equip controllers with a tip and includes a web browser-based surface providing support for interacting, handwriting, and sketching. The evaluation of the reference implementation based on the OTSS framework identified an average stylus precision of 0.98 mm (SD = 0.54 mm) and an average surface accuracy of 0.60 mm (SD = 0.32 mm) in a seated VR environment. The time for displaying the stylus movement as digital ink on the web browser surface in VR was 79.40 ms on average (SD = 23.26 ms), including the physical controller’s motion-to-photon latency visualized by its virtual representation (M = 42.57 ms, SD = 15.70 ms). The usability evaluation (N = 10) revealed a low task load, high usability, and high user experience. Participants successfully reproduced given shapes and created legible handwriting, indicating that the OTSS and it’s reference implementation is ready for everyday use. We provide source code access to our implementation, including stylus and surface calibration and surface interaction features, making it easy to reuse, extend, adapt and/or replicate previous results (https://go.uniwue.de/hci-otss).

show abstract

Section: Virtual Reality Stylusesmentioning

confidence: 99%

Section: Virtual Reality Stylusesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Off-The-Shelf Stylus: Using XR Devices for Handwriting and Sketching on Physically Aligned Virtual Surfaces

Kern

Kullmann

Ganal

et al. 2021

Front. Virtual Real.

View full text Add to dashboard Cite

show abstract

“…They explore 2D and 3D sketch in design space, analyzing these two diferent metaphors and related emerging patterns. Elsayed et al [16] showcase VRSketchPen that is a pen that provides unconstrained 3D sketching by using two haptics modalities. The frst one is a pneumatic force able to replicate a real surface constraint, and the second one is vibro-tactile feedback for emulating textures.…”

Section: Sketch In Mixed Environmentmentioning

confidence: 99%

Mixing Modalities of 3D Sketching and Speech for Interactive Model Retrieval in Virtual Reality

Giunchi

Sztrajman

James

et al. 2021

ACM International Conference on Interactive Media Experiences

View full text Add to dashboard Cite

Sketch and speech are intuitive interaction methods that convey complementary information and have been independently used for 3D model retrieval in virtual environments. While sketch has been shown to be an efective retrieval method, not all collections are easily navigable using this modality alone. We design a new challenging database for sketch comprised of 3D chairs where each of the components (arms, legs, seat, back) are independently colored. To overcome this, we implement a multimodal interface for querying 3D model databases within a virtual environment. We base the sketch on the state-of-the-art for 3D Sketch Retrieval, and use a Wizard-of-Oz style experiment to process the voice input. In this way, we avoid the complexities of natural language processing which frequently requires fne-tuning to be robust. We conduct two user studies and show that hybrid search strategies emerge from the combination of interactions, fostering the advantages provided by both modalities. CCS CONCEPTS• Human-centered computing → HCI design and evaluation methods; Virtual reality; • Computing methodologies → Object recognition; • Information systems → Retrieval models and ranking.

show abstract

“…They explore 2D and 3D sketch in design space, analyzing these two different metaphors and related emerging patterns. Elsayed et al [16] showcase VRSketchPen that is a pen that provides unconstrained 3D sketching by using two haptics modalities. The first one is a pneumatic force able to replicate a real surface constraint, and the second one is vibro-tactile feedback for emulating textures.…”

Section: Sketch In Mixed Environmentmentioning

confidence: 99%

3D sketching for interactive model retrieval in virtual reality

Giunchi

James

Steed

2018

Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic

View full text Add to dashboard Cite

show abstract

VRSketchPen: Unconstrained Haptic Assistance for Sketching in Virtual 3D Environments

Cited by 33 publications

References 56 publications

Off-The-Shelf Stylus: Using XR Devices for Handwriting and Sketching on Physically Aligned Virtual Surfaces

Off-The-Shelf Stylus: Using XR Devices for Handwriting and Sketching on Physically Aligned Virtual Surfaces

Mixing Modalities of 3D Sketching and Speech for Interactive Model Retrieval in Virtual Reality

3D sketching for interactive model retrieval in virtual reality

Contact Info

Product

Resources

About