On-the-fly hand detection training with application in egocentric action recognition

Abstract

We propose a novel approach to segment hand regions in egocentric video that requires no manual labeling of training samples. The user wearing a head-mounted camera is prompted to perform a simple gesture during an initial calibration step. A combination of color and motion analysis that exploits knowledge of the expected gesture is applied on the calibration video frames to automatically label hand pixels in an unsupervised fashion. The hand pixels identified in this manner are used to train a statisticalmodel-based hand detector. Superpixel region growing is used to perform segmentation refinement and improve robustness to noise. Experiments show that our hand detection technique based on the proposed on-the-fly training approach significantly outperforms state-of-the-art techniques with respect to accuracy and robustness on a variety of challenging videos. This is due primarily to the fact that training samples are personalized to a specific user and environmental conditions. We also demonstrate the utility of our hand detection technique to inform an adaptive video sampling strategy that improves both computational speed and accuracy of egocentric action recognition algorithms. Finally, we offer an egocentric video dataset of an insulin self-injection procedure with action labels and hand masks that can serve towards future research on both hand detection and egocentric action recognition.

Analysis of the Hands in Egocentric Vision: A Survey

“…Usually, methods for online hand segmentation made assumptions on the hand motion [55], [56], [57], [58] and/or required the user to perform a calibration with pre-defined hand movements [59]. In this way, the combination of color and motion features facilitates the detection of hand pixels, in order to train segmentation models online.…”

Head-mounted gesture controlled interface for human-computer interaction

“…Another approach dealing with hand detection and tracking in ego-centric data is [4]. Hand detection and action recognition using the data from Google glasses is considered in [17]. Finally hand pose estimation using a wearable RGB-D camera is the topic of [32] that exploits synthetic training examples and multi-class rejection-cascade classifiers.…”

Coarse-to-fine online learning for hand segmentation in egocentric video

“…These methods reduce false-positive rate of hand segmentation but also needs the offline training which requires manual labeled data. Kumar et al [10] illustrate an on-the-fly hand detection training method which is initialized by a calibration gesture performed by the user. This simple preprocessing step saves a great deal of Fig.…”

“…But, the assumption fails in many situations in which the hand is not used, such as before or after the human-computer interaction. Subsequently, some cascade detection methods are put forwarded to get rid of the assumption by checking out hand presence before performing pixel-by-pixel classification [8][9][10]. However, these approaches rely on the existence of a large training set containing a broad variety of data which are collected from multiple users under diverse illumination conditions.…”