Coarse-to-fine online learning for hand segmentation in egocentric video

Abstract

Hand segmentation is one of the most fundamental and crucial steps for egocentric human-computer interaction. The special egocentric view brings new challenges to hand segmentation tasks, such as the unpredictable environmental conditions. The performance of traditional hand segmentation methods depend on abundant manually labeled training data. However, these approaches do not appropriately capture the whole properties of egocentric human-computer interaction for neglecting the user-specific context. It is only necessary to build a personalized hand model of the active user. Based on this observation, we propose an online-learning hand segmentation approach without using manually labeled data for training. Our approach consists of top-down classifications and bottom-up optimizations. More specifically, we divide the segmentation task into three parts, a frame-level hand detection which detects the presence of the interactive hand using motion saliency and initializes hand masks for online learning, a superpixel-level hand classification which coarsely segments hand regions from which stable samples are selected for next level, and a pixel-level hand classification which produces a fine-grained hand segmentation. Based on the pixel-level classification result, we update the hand appearance model and optimize the upper layer classifier and detector. This online-learning strategy makes our approach robust to varying illumination conditions and hand appearances. Experimental results demonstrate the robustness of our approach.

Analysis of the Hands in Egocentric Vision: A Survey

“…Usually, methods for online hand segmentation made assumptions on the hand motion [55], [56], [57], [58] and/or required the user to perform a calibration with pre-defined hand movements [59]. In this way, the combination of color and motion features facilitates the detection of hand pixels, in order to train segmentation models online.…”

“…These matches were estimated using RANSAC [61] and after being removed, those left were assumed to belong to the hands and used to locate the seed point for region growing. Zhao et al [57], [58] based their approach on the typical motion pattern during actions involving the hands: preparatory phase (i.e., the hands move from the lower part of the frame to the image center) and interaction phase. During the preparatory phase they used a motion-based segmentation, computing the TV-L1 optical flow [62].…”

“…In these cases, a pre-filtering step that prevents from processing frames without any hands is necessary. This approach allows determining whether an image contains hands and it is usually followed by a hand segmentation step responsible for locating the hand region [9], [29], [32], [57], [58].…”

“…To solve this issue, the authors proposed a dynamic Bayesian network (DBN) to smooth the classification results of the SVM and improve the prediction performance [72]. Zhao et al [57], [58] detected the presence of hands within each frame exploiting the typical interaction cycle of the hands (i.e., preparatory phase -interaction -hands out of the frame). Based on this observation, they defined an ego-saliency metric related to the probability of having hands within a frame.…”

Cluster-wise learning network for multi-person pose estimation

“…Beside body joints, keypoints can be extended to refer to the small visual units with semantic information indicating the compositions, shapes and poses of the target objects, such as finger joints or key positions of any other objects. Therefore, accurate keypoint detection in unconstrained environments brings benefit to other more detailed visual understanding tasks, including semantic segmentation[1, 2, 3], saliency object segmentation [4,5,6], hand segmentation [7] and pose estimation [8,9], viewpoint estimation [10,11,12,13], salient object detection [14,15,16], attention prediction [17] and 3D reconstruction [18,19,20]. Similar as many computer vision tasks, the progress on human pose estimation problem is significantly improved by deep convolutional neural networks.…”