Investigating the Dexterity of Multi-Finger Input for Mid-Air Text Entry

Abstract

Figure 1. We investigate the dexterity of using multiple fingers for mid-air input. The paper reports performance and individuation characteristics of fingers and deploys them to the design of a mid-air text entry method using multi-objective optimization. Here we show an example of the word 'hand' being typed using one of our automatically obtained designs. ABSTRACTThis paper investigates an emerging input method enabled by progress in hand tracking: input by free motion of fingers. The method is expressive, potentially fast, and usable across many settings as it does not insist on physical contact or visual feedback. Our goal is to inform the design of high-performance input methods by providing detailed analysis of the performance and anatomical characteristics of finger motion. We conducted an experiment using a commercially available sensor to report on the speed, accuracy, individuation, movement ranges, and individual differences of each finger. Findings show differences of up to 50% in movement times and provide indices quantifying the individuation of single fingers. We apply our findings to text entry by computational optimization of multi-finger gestures in mid-air. To this end, we define a novel objective function that considers performance, anatomical factors, and learnability. First investigations of one optimization case show entry rates of 22 words per minute (WPM). We conclude with a critical discussion of the limitations posed by human factors and performance characteristics of existing markerless hand trackers.

Neuromechanics of a Button Press

“…We simulated presses with four common button types (linear, tactile, touch, mid-air). We report simulation results for (1) displacement-velocity patterns, (2) temporal precision and success rate in button activation, and (3) use of force, comparing with effects reported in empirical studies [7,36,42,46,48,53,54,59,66,69]. Over the simulations, we find evidence for the plausibility of the optimality assumption.…”

GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB

“…Estimating the 3D pose of the hand is a long-standing goal in computer vision with many applications such as in virtual/augmented reality (VR/AR) [21,31] and humancomputer interaction [43,23]. While there is a large body of existing works that consider marker-free image-based hand tracking or pose estimation, many of them require depth cameras [39,52,44,49,32,9,59] or multi-view setups [46,2,61].…”

TiTAN

“…Shape writing is easy to learn and exhibits advantages when writing English words but not with abbreviation, email addresses or passwords. Sridhar et al [20] explore chording in midair using a Leap Motion but only measured the peak performance. Unfortunately, chording is not commonly understood and requires more learning.…”

“…Midair interaction [1,5,7,12,14,17,18,20,22] is an emerging input modality for HCI, in particular for scenarios involving public displays [11,18], large surface [15], augmented and virtual reality [1] and sterile conditions (e.g., surgery). However, midair text entry has not received as much attention with most solutions requiring either the use of an external controller [1,6,14] or reflective markers [12].…”