Performance Envelopes of Virtual Keyboard Text Input Strategies in Virtual Reality

Abstract

Virtual and Augmented Reality deliver engaging interaction experiences that can transport and extend the capabilities of the user. To ensure these paradigms are more broadly usable and effective, however, it is necessary to also deliver many of the conventional functions of a smartphone or personal computer. It remains unclear how conventional input tasks, such as text entry, can best be translated into virtual and augmented reality. In this paper we examine the performance potential of four alternative text entry strategies in virtual reality (VR). These four strategies are selected to provide full coverage of two fundamental design dimensions: i) physical surface association; and ii) number of engaged fingers. Specifically, we examine typing with index fingers on a surface and in mid-air and typing using all ten fingers on a surface and in mid-air. The central objective is to evaluate the human performance potential of these four typing strategies without being constrained by current tracking and statistical text decoding limitations. To this end we introduce an auto-correction simulator that uses knowledge of the stimulus to emulate statistical text decoding within constrained experimental parameters and use high-precision motion tracking hardware to visualise and detect fingertip interactions. We find that alignment of the virtual keyboard with a physical surface delivers significantly faster entry rates over a mid-air keyboard. Also, users overwhelmingly fail to effectively engage all ten fingers in mid-air typing, resulting in slower entry rates and higher error rates compared to just using two index fingers. In addition to identifying the envelopes of human performance for the four strategies investigated, we also provide a detailed analysis of the underlying features that distinguish each strategy in terms of its performance and behaviour.

WallTokens: Surface Tangibles for Vertical Displays

“…These results cannot be generalized to average users. However, they give an indication of the performance envelope [11], i.e., the performance that expert users can reach.…”

Investigating the Performance of Gesture-Based Input for Mid-Air Text Entry in a Virtual Environment: A Comparison of Hand-Up versus Hand-Down Postures

“…In addition, previous works usually investigated the efficiency of the entry method and reported participants’ subjective satisfaction [ 21 , 22 ], and the performance of the methods in VR is evaluated by various factors, such as typing speed and ease of use [ 23 ]. The related papers paid much attention to the typing speed but ignored the user experience of participants.…”

Decoding Surface Touch Typing from Hand-Tracking

“…People type faster on a physical keyboard [3] than a soft keyboard and also faster on a surface [4] than in the air. In our study we compromise between accessibility and speed by requiring users to type against a surface but eschewing the use of a physical keyboard or a touchpad.…”

“…In our study we compromise between accessibility and speed by requiring users to type against a surface but eschewing the use of a physical keyboard or a touchpad. Additionally, surface typing has been shown to be more comfortable than mid air typing [4] and confers haptic feedback, which enhances presence in VR [16]. Our investigation caters particularly to the fastest typists-touch typists, which we mean as those who type without the sense of sight to find the keys.…”

“…We are inspired by the work of Dudley and colleagues [4] that shows the high potential of human typing efficiency and error rate given a text decoding oracle with knowledge of the text being typed. In this work, we take the first steps to reducing this oracle to practice by building a neural model of text decoding that combines motion modeling of fingers and a state-of-theart language model.…”