Sensing techniques for tablet+stylus interaction

Abstract

We explore grip and motion sensing to afford new techniques that leverage how users naturally manipulate tablet and stylus devices during pen + touch interaction. We can detect whether the user holds the pen in a writing grip or tucked between his fingers. We can distinguish bare-handed inputs, such as drag and pinch gestures produced by the nonpreferred hand, from touch gestures produced by the hand holding the pen, which necessarily impart a detectable motion signal to the stylus. We can sense which hand grips the tablet, and determine the screen's relative orientation to the pen. By selectively combining these signals and using them to complement one another, we can tailor interaction to the context, such as by ignoring unintentional touch inputs while writing, or supporting contextually-appropriate tools such as a magnifier for detailed stroke work that appears when the user pinches with the pen tucked between his fingers. These and other techniques can be used to impart new, previously unanticipated subtleties to pen + touch interaction on tablets.

Finding Common Ground

“…Touch sensing on surfaces other than displays has been investigated to create more expressive computing devices and peripherals, including mice [210], keyboards [13,45], tablet PCs [88], pens [192], and tangibles [52,68,118]. Sensors distributed around the screen can help to avoid visual occlusion [139] or to enable interaction on near-eye displays [130].…”

“…To distinguish between commonly-used grasps, a relatively large number of sensors is often needed. Either highresolution active shunt mode grids [88,131,192,199,210,229] or a multitude of loading mode sensors [33,107] deliver enough information to determine grasp. Some specific applications have used fewer sensors, for example to adapt the screen rotation depending on the user's grip [32], to start an application [118], or to sense a small number of different grasps through only a few sensors [223].…”

“…Inertial measurement units (IMU) are often paired with capacitive systems in order to reveal more contextual information [65,88], or to provide an additional dimension for user input [114,130,209]. IMUs have also been used to correlate whole-body movements to capacitive sensing data [109,194] and avoid accidental interactions [107].…”

DeepWriting

“…Given the naturalness of the medium [47,11], pen-based interfaces have seen enduring interest in both the graphics and HCI literature [50]. Ever since Ivan Sutherland's Sketchpad [51] researchers have explored sensing and input techniques for small screens [27,57], tablets [23,40] and whiteboards [36,39,56] and have proposed ways of integrating paper with digital media [21,6]. Furthermore many domain specific applications have been proposed.…”

“…In this paper we explore novel ways to combine the benefits of digital ink with the versatility and efficiency of digital text, by making it editable via disentanglement of style and content. Digital ink and stylus-based interfaces have been of continued interest to HCI research (e.g., [22,23,44,56,61]). However, to process digital ink one has typically to resort to optical character recognition (OCR) techniques (e.g., [17]) thus invariably losing the personalized aspect of written text.…”