Viewpoint Integration for Hand-Based Recognition of Social Interactions from a First-Person View

Abstract

Wearable devices are becoming part of everyday life, from first-person cameras (GoPro, Google Glass), to smart watches (Apple Watch), to activity trackers (FitBit). These devices are often equipped with advanced sensors that gather data about the wearer and the environment. These sensors enable new ways of recognizing and analyzing the wearer’s everyday personal activities, which could be used for intelligent human-computer interfaces and other applications. We explore one possible application by investigating how egocentric video data collected from head-mounted cameras can be used to recognize social activities between two interacting partners (e.g. playing chess or cards). In particular, we demonstrate that just the positions and poses of hands within the first-person view are highly informative for activity recognition, and present a computer vision approach that detects hands to automatically estimate activities. While hand pose detection is imperfect, we show that combining evidence across first-person views from the two social partners significantly improves activity recognition accuracy. This result highlights how integrating weak but complimentary sources of evidence from social partners engaged in the same task can help to recognize the nature of their interaction.

Analysis of the Hands in Egocentric Vision: A Survey

“…The effort of many researchers in finding novel and powerful approaches to obtain better results is justified by the possibility to improve not only the localization accuracy, but also to boost the performance of higher-level inference. In fact, it was demonstrated that a good hand segmentation mask can be sufficient for recognizing actions and activities involving the hands with high accuracy [16], [71]. For this reason, pixellevel segmentation has often been used as basis of higher-inference methods.…”

“…Many authors proposed regionbased CNNs to detect the hands, exploiting segmentation approaches (Section 3.1) to generate region proposals. Bambach et al [16], [71] proposed a probabilistic approach for region proposal generation that combined spatial biases (e.g., reasoning on the position of the shape of the hands from training data) and appearance models (e.g., non-parametric modeling of skin color in the YUV color space). To guarantee high coverage, they generated 2,500 regions for each frame that were classified using CaffeNet [75].…”

“…In fact, some of the methods described in the previous subsections were also tested within an activity recognition framework [67], [138]. Generally, we can identify two types of approaches: activity recognition based on actions and interactions [3], [78] and approaches that used hand localization results to directly prectict the activities [16], [46], [71].…”

Identifying First-Person Camera Wearers in Third-Person Videos

“…Several recent papers have shown the potential for combining first-person video analysis with evidence from other types of synchronized video, including from other firstperson cameras [3,29], multiple third-person cameras [26], or even hand-mounted cameras [5]. However, these papers assume that a single person appears in each video, avoiding the person-level correspondence problem.…”

“…Despite its importance, we are aware of very little work that tries to address this problem. Several recent papers propose using multiple cameras for joint first-person recognition [3,5,26,29], but make simplistic assumptions like that only one person appears in the scene. Using visual SLAM to infer first-person camera trajectory and map to third-person cameras (e.g., [17,19]) works well in some settings, but can fail for crowded environments when longterm precise localizations are needed and when first-person video has significant motion blur.…”