A three-step interaction pattern for improving discoverability in finger identification techniques

Abstract

Figure 1: Our three-application demo: (a) SMS application: the thumb can be used to feed-forward the current mapping between indivdual fingers and commands; here, the index triggers the copy command. (b) PDF annotation application: (b1) thumb showing the different subsets of commands available for selection; after sliding the thumb in the corresponding direction, (b2) updated feed-forward corresponding to the top-right set. (c) Vectorial drawing application demo: non-dominant thumb and index select a command mapping enabling the dominant ring finger to draw an ellipse. ABSTRACTIdentifying which fingers are in contact with a multi-touch surface provides a very large input space that can be leveraged for command selection. However, the numerous possibilities enabled by such vast space come at the cost of discoverability. To alleviate this problem, we introduce a three-step interaction pattern inspired by hotkeys that also supports feedforward. We illustrate this interaction with three applications allowing us to explore and adapt it in different contexts.

Glass+Skin: An Empirical Evaluation of the Added Value of Finger Identification to Basic Single-Touch Interaction on Touch Screens

“…In this class of interaction, proper finger identificationwith screen and skin coordinates jointly taken into accountseems highly promising [1,6,9,11,12,18,23,26,37,40]. Many studies have concentrated on triggering fingerdependent commands or action [1,6,23,26,37].…”

“…Many studies have concentrated on triggering fingerdependent commands or action [1,6,23,26,37]. For instance, Adoiraccourcix maps different modifiers to the fingers of the non-dominant hand and different commands to fingers of the dominant hand [12]. Finger identification can also be coupled with physical buttons as in recent Apple Smartphones [35,40].…”

“…Another approach consists of analyzing EMG signals on the forearm to determine which finger is applying pressure to the surface [6]. In yet another approach, Goguet et al attached GameTraks 1 to user's fingers [11,12]. Of course, digital gloves can also serve to track user fingers [34].…”

WhichFingers

“…Other prototypes developed in the literature are mainly based on contacts geometry [1,5,9,31,18,29], RGB and depth cameras [4,11,13,17,24,30,33], and fingerprints [27,16].…”

“…RGB cameras have been used to recognize specific hand postures involving different fingers [17,21]. Depth cameras can help to segment the hand from the background [24] but as soon as the hand can interact more freely, fingers need to be equipped with color rings to improve the robustness of detection [11,13,30]. Leap Motion has been investigated but it requires users to first open the hand such that the algorithm identifies the 5 fingers, and then fold the fingers such that the finger selected for interaction is the furthest away from the others [4].…”