Foreground-Aware Stylization and Consensus Pseudo-Labeling for Domain Adaptation of First-Person Hand Segmentation

Abstract

Hand segmentation is a crucial task in first-person vision. Since first-person images exhibit strong bias in appearance among different environments, adapting a pre-trained segmentation model to a new domain is required in hand segmentation. Here, we focus on appearance gaps for hand regions and backgrounds separately. We propose (i) foreground-aware image stylization and (ii) consensus pseudolabeling for domain adaptation of hand segmentation. We stylize source images independently for the foreground and background using target images as style. To resolve the domain shift that the stylization has not addressed, we apply careful pseudo-labeling by taking a consensus between the models trained on the source and stylized source images. We validated our method on domain adaptation of hand segmentation from real and simulation images. Our method achieved state-of-the-art performance in both settings. We also demonstrated promising results in challenging multi-target domain adaptation and domain generalization settings.

Efficient Annotation and Learning for 3D Hand Pose Estimation: A Survey

“…As shown in Fig. 3, generating data using synthetic models [8,23,37,39,65] is costeffective, but it crates unrealistic hand texture [41]. Although the hand-marker-based annotation [15,55,62] can automatically track the 6-DoF of sensor-attached hand joints, the sensors distort the hand appearance and hinder the natural hand movement.…”

“…Ohkawa et al . proposed foreground-aware image stylization to convert the simulation texture in the ObMan data to a more realistic one while separating the hand regions and backgrounds [41]. However, the ObMan data provide static hand images with hand-held objects but without hand motion.…”

“…This question will be taken into consideration in the development of 3D hand pose estimators because it is not straightforward to acquire 3D hand pose annotation (i.e., 3D keypoints of the hand joints) and train the estimators with limited resources in deployment. The two key issues concerning annotation and learning have inhibited us from constructing accurate 3D hand pose estimators and utilizing them in downstream applications, such as hand-object interaction analysis [2,48,57], egocentric vision [12,18,41], augmented and virtual reality [31,59], and robot learning from human demonstration [21,34,44].…”

One-Shot Learning with Pseudo-Labeling for Cattle Video Segmentation in Smart Livestock Farming

“…For example, Tokunaga et al [ 14 ] utilized pseudo-labeling and class proportion to realize semantic segmentation. Ohkawa et al [ 16 ] proposed consensus pseudo-labeling for segmenting the hand image. Zou et al [ 17 ] generated structured pseudo-labels for semantic segmentation.…”