Identifying First-Person Camera Wearers in Third-Person Videos

Abstract

We consider scenarios in which we wish to perform joint scene understanding, object tracking, activity recognition, and other tasks in environments in which multiple people are wearing body-worn cameras while a third-person static camera also captures the scene. To do this, we need to establish person-level correspondences across first-and thirdperson videos, which is challenging because the camera wearer is not visible from his/her own egocentric video, preventing the use of direct feature matching. In this paper, we propose a new semi-Siamese Convolutional Neural Network architecture to address this novel challenge. We formulate the problem as learning a joint embedding space for first-and third-person videos that considers both spatial-and motion-domain cues. A new triplet loss function is designed to minimize the distance between correct first-and third-person matches while maximizing the distance between incorrect ones. This end-to-end approach performs significantly better than several baselines, in part by learning the first-and third-person features optimized for matching jointly with the distance measure itself.

Visual-GPS: Ego-Downward and Ambient Video Based Person Location Association

“…Baselines We first implement multiple baselines to compare the performance considering inputs, and models. These baseline method are proposed in peer researches [14,27,23] including spatial-domain siamese network [14], motion-domain siamese network [14], twostream semi-siamese network [14], triplet network [27], and temporal domain image and flow network [14,23]. We also demonstrate the weight share performance for siamesenetwork.…”

“…The authors proposed a 'Graph' representation for temporal and spatial matching. In [14], the authors solved the task to localize the person in the third view if given the both the third and ego camera frames. In this paper, spatial-domain semisiamese, motion-domain semi-siamese, dual-domain semisiamese, and dual-domain semi-triplet networks are well studied.…”

“…Unlike the top-and-forward view [5] and third-forward view [14] cases, the ego-downward mounting faces the following challenges : 1) appearance verification across different views does not hold under this situation since it is not pointing out to scenario; 2) clothes texture verification will not work since in large crowd there should have the simi-lar dressing or occlusion; 3) the same action with different initial pose state (in world coordinate system) will also mislead the model since the ego-downward frames will not tell the difference (in Fig.3). Thus, using a general siamese or triplet model to correlate the two views with temporal and spatial information would fail [14]. Moreover, the graph solution using relative view insight will not happen under this situation [5].…”

Joint Person Segmentation and Identification in Synchronized First- and Third-Person Videos

“…Our hypothesis is that simultaneous segmentation and matching is mutually beneficial: segmentation helps refine matching by producing finer-grained appearance features (compared to bounding boxes), which are important in crowded scenes with many occlusions, while matching helps locate a person of interest and produce better segmentation masks, which in turn help in tasks like activity and action recognition. We show that previous work [14] is a special case of ours, since we can naturally handle their first-and third-person cases. We evaluate on two publicly available datasets augmented with pixel-level annotations, showing that we achieve significantly better results than numerous baselines.…”

“…That paper's approach is applicable in closed settings with overhead cameras (e.g., a museum), but not in unconstrained environments such as our law enforcement example. Fan et al [14] relax many assumptions, allowing arbitrary third-person camera views and including evidence based on scene appearance. Zheng et al [43] consider the distinct problem of identifying the same person appearing in multiple wearable camera videos (but not trying to identify the camera wearers themselves).…”