AirTouch: Synchronizing In-air Hand Gesture and On-body Tactile Feedback to Augment Mobile Gesture Interaction

Lee, Seungyon Claire; Li, Bohao; Starner﻿﻿, Thad

doi:10.1109/iswc.2011.27

Cited by 24 publications

(23 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this sense we share commonality with prior work combining pen+touch input [14], motion+touch [13], in-air and on-body gestures [25], or input from multiple devices [5]. We explore new interaction techniques and application scenarios that combine touch input on and gestural interaction around a mobile device.…”

Section: Related Workmentioning

confidence: 83%

In-air gestures around unmodified mobile devices

Song

Sörös

Pece

et al. 2014

Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology

148

View full text Add to dashboard Cite

Figure 1: Touch input is expressive but can occlude large parts of the screen (A). We propose a machine learning based algorithm for gesture recognition expanding the interaction space around the mobile device (B), adding in-air gestures and hand-part tracking (D) to commodity off-the-shelf mobile devices, relying only on the device's camera (and no hardware modifications). We demonstrate a number of compelling interactive scenarios including bi-manual input to mapping and gaming applications (C+D). The algorithm runs in real time and can even be used on ultra-mobile devices such as smartwatches (E). ABSTRACTWe present a novel machine learning based algorithm extending the interaction space around mobile devices. The technique uses only the RGB camera now commonplace on off-the-shelf mobile devices. Our algorithm robustly recognizes a wide range of in-air gestures, supporting user variation, and varying lighting conditions. We demonstrate that our algorithm runs in real-time on unmodified mobile devices, including resource-constrained smartphones and smartwatches. Our goal is not to replace the touchscreen as primary input device, but rather to augment and enrich the existing interaction vocabulary using gestures. While touch input works well for many scenarios, we demonstrate numerous interaction tasks such as mode switches, application and task management, menu selection and certain types of navigation, where such input can be either complemented or better served by inair gestures. This removes screen real-estate issues on small touchscreens, and allows input to be expanded to the 3D space around the device. We present results for recognition accuracy (93% test and 98% train), impact of memory footprint and other model parameters. Finally, we report results from preliminary user evaluations, discuss advantages and limitations and conclude with directions for future work.

show abstract

Section: Related Workmentioning

confidence: 83%

In-air gestures around unmodified mobile devices

Song

Sörös

Pece

et al. 2014

Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology

148

View full text Add to dashboard Cite

show abstract

“…Gesture interactions include simple hand movements over a device [13,14], precise selection techniques based on finger movements [5,9,19] and more subtle gestures with wearables [1]. These interfaces give users feedback in a variety of ways.…”

Section: Related Workmentioning

confidence: 99%

“…Users interacted with AirTouch [14] with hand movements over wrist-worn sensors. Each of its four sensors was paired with an actuator, giving spatial vibrotactile feedback to show which sensors detected input.…”

Section: Tactile Feedbackmentioning

confidence: 99%

Tactile Feedback for Above-Device Gesture Interfaces

Freeman

Brewster

Lantz

2014

Proceedings of the 16th International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Above-device gesture interfaces let people interact in the space above mobile devices using hand and finger movements. For example, users could gesture over a mobile phone or wearable without having to use the touchscreen. We look at how above-device interfaces can also give feedback in the space over the device. Recent haptic and wearable technologies give new ways to provide tactile feedback while gesturing, letting touchless gesture interfaces give touch feedback. In this paper we take a first detailed look at how tactile feedback can be given during above-device interaction. We compare approaches for giving feedback (ultrasound haptics, wearables and direct feedback) and also look at feedback design. Our findings show that tactile feedback can enhance above-device gesture interfaces.

show abstract

“…If two hands are used, one hand can be used to make the gesture and the other hand can provide the reference point (Gustafson 2012). Lee, Li, and Starner (2011) found that tactile feedback given on the users' wrist not only improved performance in a condition where visual feedback was available, but the tactile feedback could also compensate for the lack of visual feedback (e.g. while walking).…”

Section: Feedback For Different Input/interaction Modalitiesmentioning

confidence: 95%

Effects of auditory, haptic and visual feedback on performing gestures by gaze or by hand

Köpsel

Majaranta

Isokoski

et al. 2016

Behaviour & Information Technology

View full text Add to dashboard Cite

Modern interaction techniques like non-intrusive gestures provide means for interacting with distant displays and smart objects without touching them. We were interested in the effects of feedback modality (auditory, haptic or visual) and its combined effect with input modality on user performance and experience in such interactions. Therefore, we conducted two exploratory experiments where numbers were entered, either by gaze or hand, using gestures composed of four stroke elements (up, down, left and right). In Experiment 1, a simple feedback was given on each stroke during the motor action of gesturing: an audible click, a haptic tap or a visual flash. In Experiment 2, a semantic feedback was given on the final gesture: the executed number was spoken, coded by haptic taps or shown as text. With simultaneous simple feedback in Experiment 1, performance with hand input was slower but more accurate than with gaze input. With semantic feedback in Experiment 2, however, hand input was only slower. Effects of feedback modality were of minor importance; nevertheless, semantic haptic feedback in Experiment 2 showed to be useless at least without extensive training. Error patterns differed between both input modes, but again not dependent on feedback modality. Taken together, the results show that in designing gestural systems, choosing a feedback modality can be given a low priority; it can be chosen according to the task, context and user preferences. ARTICLE HISTORY

show abstract

AirTouch: Synchronizing In-air Hand Gesture and On-body Tactile Feedback to Augment Mobile Gesture Interaction

Cited by 24 publications

References 18 publications

In-air gestures around unmodified mobile devices

In-air gestures around unmodified mobile devices

Tactile Feedback for Above-Device Gesture Interfaces

Effects of auditory, haptic and visual feedback on performing gestures by gaze or by hand

Contact Info

Product

Resources

About