In-air gestures around unmodified mobile devices

Abstract

Figure 1: Touch input is expressive but can occlude large parts of the screen (A). We propose a machine learning based algorithm for gesture recognition expanding the interaction space around the mobile device (B), adding in-air gestures and hand-part tracking (D) to commodity off-the-shelf mobile devices, relying only on the device's camera (and no hardware modifications). We demonstrate a number of compelling interactive scenarios including bi-manual input to mapping and gaming applications (C+D). The algorithm runs in real time and can even be used on ultra-mobile devices such as smartwatches (E). ABSTRACTWe present a novel machine learning based algorithm extending the interaction space around mobile devices. The technique uses only the RGB camera now commonplace on off-the-shelf mobile devices. Our algorithm robustly recognizes a wide range of in-air gestures, supporting user variation, and varying lighting conditions. We demonstrate that our algorithm runs in real-time on unmodified mobile devices, including resource-constrained smartphones and smartwatches. Our goal is not to replace the touchscreen as primary input device, but rather to augment and enrich the existing interaction vocabulary using gestures. While touch input works well for many scenarios, we demonstrate numerous interaction tasks such as mode switches, application and task management, menu selection and certain types of navigation, where such input can be either complemented or better served by inair gestures. This removes screen real-estate issues on small touchscreens, and allows input to be expanded to the 3D space around the device. We present results for recognition accuracy (93% test and 98% train), impact of memory footprint and other model parameters. Finally, we report results from preliminary user evaluations, discuss advantages and limitations and conclude with directions for future work.

Collocated Multi-user Gestural Interactions with Unmodified Wearable Devices

“…GestureWatch [21] and HoverFlow [22] are both systems that recognize in-air hand gestures performed over mobile or wearable devices relying on infrared proximity sensors. Hand gestures can also be recognized from a single RGB camera found on most mobile devices [40]. WatchMe [41] is a camera-based system that can track a pen or a laser pointer on a drawing canvas and use this as an input modality.…”

Joint Estimation of 3D Hand Position and Gestures from Monocular Video for Mobile Interaction

“…All these approaches require the user to wear specialized sensor electronics and might not be feasible in real-world scenarios. Song et al [13] recently introduced a data-driven gesture recognition approach that enables mid-air interaction on unmodified portable devices. However, the type of interactions are limited to 2D gestures.…”

“…Our method is similar to those proposed in [2,13]. In [13] several randomized decision forests (RFs) are combined to enable the recognition of discrete 2D hand shapes or gestures.…”

“…In [13] several randomized decision forests (RFs) are combined to enable the recognition of discrete 2D hand shapes or gestures. We extend [13] to jointly detect hand shapes and to regress the average hand depth. Together with the hand's centroid this gives full 3D hand position.…”

“…To achieve this, we take a data-driven machine learning approach similar to those presented in [2,13] but extend this framework to a hybrid classification-regression scheme, capable of learning a mapping from 2D images to 3D hand positions plus gestures.…”