Direct Pen Interaction With a Conventional Graphical User Interface

Abstract

We examine the usability and performance of Tablet PC direct pen input with a conventional graphical user interface (GUI). We use a qualitative observational study design with 16 participants divided into 4 groups: 1 mouse group for a baseline control and 3 Tablet PC groups recruited according to their level of experience. The study uses a scripted scenario of realistic tasks and popular office applications designed to exercise standard GUI components and cover typical interactions such as parameter selection, object manipulation, text selection, and ink annotation. We capture a rich set of logging data including 3D motion capture, video taken from the participants' point-of-view, screen capture video, and pen events such as movement and taps. To synchronize, segment, and annotate these logs, we used our own custom analysis software.We find that pen participants make more errors, perform inefficient movements, and express frustration during many tasks. Our observations reveal overarching problems with direct pen input: poor precision when tapping and dragging, errors caused by hand occlusion, instability and fatigue due to ergonomics and reach, cognitive differences between pen and mouse usage, and frustration due to limited input capabilities. We believe these to be the primary causes of nontext errors, which contribute to user frustration when using a pen with a conventional GUI. Finally, we discuss how researchers could address these issues without sacrificing the consistency of current GUIs and applications by making improvements at three levels: hardware, base interaction, and widget behavior.

Occlusion-aware interfaces

“…An occlusion-aware dragging technique (Figure 10b) could address the fourth occlusion-related issue identified by our study of Tablet PC usability [18] by reducing inefficient movements when dragging. By using the model to detect when the user is dragging the cursor into an occluded area, the area in front of the cursor could be displayed in a nonoccluded callout.…”

“…This technique addresses three out of four issues we identified in our study [18], and provides a case study of related research problems when developing occlusion-aware techniques. The technique displays occluded regions in a bubble-like callout.…”

“…We conducted a study of Tablet PC usability with realistic tasks and common software applications [18] and found occlusion likely contributed to error and fatigue:…”

Hand occlusion on a multi-touch tabletop

“…Vogel and Balakrishnan [22] list hand occlusion as one of five direct pen input problems. They observed hidden status messages, missed previews, inefficient movements, and occlusion contortion.…”

“…This can be a problem, because compared to manipulating objects on a real tabletop, a tabletop computer is dynamic and can display relevant information, sequential widgets, and system messages in occluded areas. Researchers are aware of occlusion: they suggest it impedes performance [9,21,22] and use it to motivate the design of interaction techniques [7,13,15,20,24]. Yet, there has not been a systematic study of hand occlusion with multi-touch tabletops.…”