Teaching motion gestures via recognizer feedback

Abstract

When using motion gestures, 3D movements of a mobile phone, as an input modality, one significant challenge is how to teach end users the movement parameters necessary to successfully issue a command. Is a simple video or image depicting movement of a smartphone sufficient? Or do we need three-dimensional depictions of movement on external screens to train users? In this paper, we explore mechanisms to teach end users motion gestures, examining two factors. The first factor is how to represent motion gestures: as icons that describe movement, video that depicts movement using the smartphone screen, or a Kinect-based teaching mechanism that captures and depicts the gesture on an external display in three-dimensional space. The second factor we explore is recognizer feedback, i.e. a simple representation of the proximity of a motion gesture to the desired motion gesture based on a distance metric extracted from the recognizer. We show that, by combining video with recognizer feedback, participants master motion gestures equally quickly as end users that learn using a Kinect. These results demonstrate the viability of training end users to perform motion gestures using only the smartphone display.

Using audio cues to support motion gesture interaction on mobile devices

“…Recent work by Kamal et al [19] explored the effect of using various gesture representation systems, with and without visual feedback, on user performance of motion gestures.…”

“…The control technique was designed in this way since, at the time this research was conducted, there were no other training techniques for motion gestures that did not additional hardware (such as a Kinect [19]). …”

Effect of Motion-Gesture Recognizer Error Pattern on User Workload and Behavior

“…IUI 2017, March 13 -16, 2017, Limassol, Cyprus gestures input from everyday movement [31], teaching appropriate movements necessary to invoke commands [17], and designing reliable recognizers [25].…”

Touch, Movement and Vibration: User Perception of Vibrotactile Feedback for Touch and Mid-Air Gestures

“…Related research has shown that delivering appropriate feedback to users during gesture input can help with gesture training [7], increase recognition accuracy [8], and improve overall user experience [5]. However, providing feedback beyond visual and audio cues is still subject of technical development and investigation of user perception of vibrotactile stimuli [1,9,11].…”

“…These new devices are worn on various body parts and communicate with the user's smartphone, which is emerging as the central unit of a distributed on-body network of devices [2]. Although prior art has investigated vibrotactile feedback delivered on the smartphone itself [7], no work has examined users' perceptions of feedback decoupled from the smartphone and delivered on the body by an accessory wearable during gesture input with the primary device.…”