ReconViguRation: Reconfiguring Physical Keyboards in Virtual Reality

Schneider, Daniel; Otte, Alexander; Gesslein, Travis; Gagel, Philipp; Kuth, Bastian; Damlakhi, Mohamad Shahm; Dietz, Oliver; Ofek, Eyal; Pahud, Michel; Kristensson, Per Ola; Müller, Jörg; Grubert, Jens

doi:10.1109/tvcg.2019.2932239

Cited by 53 publications

(42 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Text entry is perhaps one of the most challenging input tasks in virtual and augmented reality and many different text entry methods have been investigated [3]. While researchers have studied appropriating physical keyboards for VR [19,27,43], a likely future dominant solution is the transplanting of full-sized keyboards into VR and AR environments as full-sized virtual keyboards available for either mid-air typing or surface-aligned typing supported by a reappropriated nearby hard surface. However, to support such typing is highly challenging, as the targets (keys) are tightly packed, requiring highly precise hand and finger tracking [40], as well as strong predictive models to account for incorrect key hits.…”

Section: Related Workmentioning

confidence: 99%

Complex Interaction as Emergent Behaviour: Simulating Mid-Air Virtual Keyboard Typing using Reinforcement Learning

Hetzel

Dudley

Feit

et al. 2021

IEEE Trans. Visual. Comput. Graphics

Self Cite

View full text Add to dashboard Cite

Accurately modelling user behaviour has the potential to significantly improve the quality of human-computer interaction. Traditionally, these models are carefully hand-crafted to approximate specific aspects of well-documented user behaviour. This limits their availability in virtual and augmented reality where user behaviour is often not yet well understood. Recent efforts have demonstrated that reinforcement learning can approximate human behaviour during simple goal-oriented reaching tasks. We build on these efforts and demonstrate that reinforcement learning can also approximate user behaviour in a complex mid-air interaction task: typing on a virtual keyboard. We present the first reinforcement learning-based user model for mid-air and surface-aligned typing on a virtual keyboard. Our model is shown to replicate high-level human typing behaviour. We demonstrate that this approach may be used to augment or replace human testing during the validation and development of virtual keyboards.

show abstract

Section: Related Workmentioning

confidence: 99%

Complex Interaction as Emergent Behaviour: Simulating Mid-Air Virtual Keyboard Typing using Reinforcement Learning

Hetzel

Dudley

Feit

et al. 2021

IEEE Trans. Visual. Comput. Graphics

Self Cite

View full text Add to dashboard Cite

show abstract

“…Büschel et al [15] surveyed a wide range of immersive interaction techniques for visual analytics. Previous research on productivity desktop-based VR has concentrated on the use of physical keyboards [99], controllers and hands [55,133], and, recently, tablets [111]. The closest work to this paper is by Surale et al [111], which focuses on spatial manipulation and creation of 3D objects.…”

Section: Mixed Reality For Knowledge Workmentioning

confidence: 99%

Breaking the Screen: Interaction Across Touchscreen Boundaries in Virtual Reality for Mobile Knowledge Workers

Biener

Schneider

Gesslein

et al. 2020

IEEE Trans. Visual. Comput. Graphics

Self Cite

View full text Add to dashboard Cite

Virtual Reality (VR) has the potential to transform knowledge work. One advantage of VR knowledge work is that it allows extending 2D displays into the third dimension, enabling new operations, such as selecting overlapping objects or displaying additional layers of information. On the other hand, mobile knowledge workers often work on established mobile devices, such as tablets, limiting interaction with those devices to a small input space. This challenge of a constrained input space is intensified in situations when VR knowledge work is situated in cramped environments, such as airplanes and touchdown spaces. In this paper, we investigate the feasibility of interacting jointly between an immersive VR head-mounted display and a tablet within the context of knowledge work. Specifically, we 1) design, implement and study how to interact with information that reaches beyond a single physical touchscreen in VR; 2) design and evaluate a set of interaction concepts; and 3) build example applications and gather user feedback on those applications.

show abstract

“…The work of Schneider et al 1 could also be applied to augmented reality (AR), which might become an even more ubiquitous technology for productivity work. One can certainly imagine future workers replacing their laptops with AR glasses and a lightweight, wireless keyboard, allowing them to work from anywhere without sacrificing screen real estate.…”

Section: Spotlight On Transactionsmentioning

confidence: 99%

“…In "ReconViguRation: Reconfiguring Physical Keyboards in Virtual Reality," Daniel Schneider and colleagues 1 show that, in VR, physical keyboards can become much more than a device for entering characters. Their article, which also won the best paper award at IEEE ISMAR 2019, proposes that physical keyboards can be adapted for use in a variety of VR applications and interactions capable of taking advantage of their unique affordances (for example, dozens of unique keys, haptic feedback for both touching and depressing the keys, and broad familiarity…”

mentioning

confidence: 99%