We propose a new approach for improving text entry accuracy on touchscreen keyboards by adapting the underlying spatial model to factors such as input hand postures, individuals, and target key positions. To combine these factors together, we introduce a hierarchical spatial backoff model (SBM) that consists of submodels with different levels of complexity. The most general model includes no adaptive factors, whereas the most specific model includes all three. Considering that in practice people may switch hand postures (e.g., from twothumb to one-finger) to better suit a situation, and that the specific submodels may take time to train for each user, a specific submodel should be applied only if its corresponding input posture can be identified with confidence, and if the submodel has enough training data from the user. We introduce the backoff mechanism to fall back to a simpler model if either of these conditions are not met. We implemented a prototype system capable of reducing the language-modelindependent error rate by 13.2% using an online posture classifier with 86.4% accuracy. Further improvements in error rate may be possible with even better posture classification.

Modern smartphones correct typing errors and learn userspecific words (such as proper names). Both techniques are useful, yet little has been published about their technical specifics and concrete benefits.One reason is that typing accuracy is difficult to measure empirically on a large scale. We describe a closed-loop, smart touch keyboard (STK) evaluation system that we have implemented to solve this problem. It includes a principled typing simulator for generating human-like noisy touch input, a simple-yet-effective decoder for reconstructing typed words from such spatial data, a large web-scale background language model (LM), and a method for incorporating LM personalization. Using the Enron email corpus as a personalization test set, we show for the first time at this scale that a combined spatial/language model reduces word error rate from a pre-model baseline of 38.4% down to 5.7%, and that LM personalization can improve this further to 4.6%.

We present an extension to a mobile keyboard input decoder based on finite-state transducers that provides general transliteration support, and demonstrate its use for input of South Asian languages using a QWERTY keyboard. On-device keyboard decoders must operate under strict latency and memory constraints, and we present several transducer optimizations that allow for high accuracy decoding under such constraints. Our methods yield substantial accuracy improvements and latency reductions over an existing baseline transliteration keyboard approach. The resulting system was launched for 22 languages in Google Gboard in the first half of 2017.

Personal user-defined gesture shortcuts have shown great potential for accessing the ever-growing amount of data and computing power on touchscreen mobile devices. However, their lack of scalability is a major challenge for their wide adoption. In this paper, we present Gesture Marks, a novel approach to touch-gesture interaction that allows a user to access applications and websites using gestures without having to define them first. It offers two distinctive solutions to address the problem of scalability. First, it leverages the "wisdom of the crowd", a continually evolving library of gesture shortcuts that are collected from the user population, to infer the meaning of gestures that a user never defined himself. Second, it combines an extensible template-based gesture recognizer with a specialized handwriting recognizer to even better address handwriting-based gestures, which are a common form of gesture shortcut. These approaches effectively bootstrap a user's personal gesture library, alleviating the need to define most gestures manually. Our work was motivated and validated via a series of user studies, and the findings from these studies add to the body of knowledge on gesture-based interaction.

We present WatchWriter, a finger operated keyboard that supports both touch and gesture typing with statistical decoding on a smartwatch. Just like on modern smartphones, users type one letter per tap or one word per gesture stroke on WatchWriter but in a much smaller spatial scale. Watch-Writer demonstrates that human motor control adaptability, coupled with modern statistical decoding and error correction technologies developed for smartphones, can enable a surprisingly effective typing performance despite the small watch size. In a user performance experiment entirely run on a smartwatch, 36 participants reached a speed of 22-24 WPM with near zero error rate.