Head Motion Signatures from Egocentric Videos

Poleg, Yair; Arora, Chetan; Peleg, Bezalel

doi:10.1007/978-3-319-16811-1_21

Cited by 25 publications

(25 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 1 clearly indicates that our approach of learning the shared embedding space for first-and third-person videos significantly outperforms the baselines. Unlike previous work relying on classic hand-crafted features like head trajectories (e.g., [19]), our method learns the optimal embedding representation from training data in an endto-end fashion, yielding a major increase in accuracy. We also compared our Siamese and semi-Siamese architectures against the model of not sharing any layers between firstperson and third-person branches (Not-Siamese in Table 1), showing that semi-Siamese yields better accuracy.…”

Section: Resultsmentioning

confidence: 99%

“…We believe this is the first paper to formulate first-and third-person video correspondence as an embedding space learning problem and to present an end-to-end learning approach. Unlike previous work [19,28] which uses handcoded trajectory features to match videos without any embedding learning, our method is applicable in more complex environments (e.g. with arbitrarily placed first-and thirdperson cameras and arbitrary numbers of people).…”

Section: Related Workmentioning

confidence: 99%

“…In particular, the latter baselines have exactly the same loss function as ours by using fully connected layers. Finally, we implemented Poleg et al's head motion signatures [19], which track bounding boxes of people in third-person frames and correlates them with average XY optical flows in first-person frames. Figure 3 presents precision-recall curves for variants of our technique and the baselines, and Table 1 summarizes in terms of Average Precision (AP).…”

Section: Baselinesmentioning

confidence: 99%

“…Several recent papers propose using multiple cameras for joint first-person recognition [3,5,26,29], but make simplistic assumptions like that only one person appears in the scene. Using visual SLAM to infer first-person camera trajectory and map to third-person cameras (e.g., [17,19]) works well in some settings, but can fail for crowded environments when longterm precise localizations are needed and when first-person video has significant motion blur. Ardeshir and Borji [2] match a set of egocentric videos to people appearing in a top-view video using graph matching, but assume that there are multiple first-person cameras sharing the same field of view at any given time, and only consider purely overhead third-person cameras (not oblique or ground-level views).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Identifying First-Person Camera Wearers in Third-Person Videos

Cheng

Lee

et al. 2017

2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

We consider scenarios in which we wish to perform joint scene understanding, object tracking, activity recognition, and other tasks in environments in which multiple people are wearing body-worn cameras while a third-person static camera also captures the scene. To do this, we need to establish person-level correspondences across first-and thirdperson videos, which is challenging because the camera wearer is not visible from his/her own egocentric video, preventing the use of direct feature matching. In this paper, we propose a new semi-Siamese Convolutional Neural Network architecture to address this novel challenge. We formulate the problem as learning a joint embedding space for first-and third-person videos that considers both spatial-and motion-domain cues. A new triplet loss function is designed to minimize the distance between correct first-and third-person matches while maximizing the distance between incorrect ones. This end-to-end approach performs significantly better than several baselines, in part by learning the first-and third-person features optimized for matching jointly with the distance measure itself.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Baselinesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Identifying First-Person Camera Wearers in Third-Person Videos

Cheng

Lee

et al. 2017

2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…To the best of our knowledge, most of the first-person video datasets comprise scenes where only a limited number of people are observed, e.g., CMU Social Interaction Dataset [27], JPL Interaction Dataset [34], HUJI EgoSeg Dataset [29]. In this work, we introduce a new dataset which we call First-Person Locomotion (FPL) Dataset.…”

Section: First-person Locomotion Datasetmentioning

confidence: 99%

Future Person Localization in First-Person Videos

Yagi

Mangalam

Yonetani

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

170

145

View full text Add to dashboard Cite

We present a new task that predicts future locations of people observed in first-person videos. Consider a firstperson video stream continuously recorded by a wearable camera. Given a short clip of a person that is extracted from the complete stream, we aim to predict that person's location in future frames. To facilitate this future person localization ability, we make the following three key observations: a) First-person videos typically involve significant ego-motion which greatly affects the location of the target person in future frames; b) Scales of the target person act as a salient cue to estimate a perspective effect in first-person videos; c) First-person videos often capture people up-close, making it easier to leverage target poses (e.g., where they look) for predicting their future locations. We incorporate these three observations into a prediction framework with a multi-stream convolution-deconvolution architecture. Experimental results reveal our method to be effective on our new dataset as well as on a public social interaction dataset.

show abstract

In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video

Liu

Rehg

2018

Lecture Notes in Computer Science

224

296

View full text Add to dashboard Cite

We address the task of jointly determining what a person is doing and where they are looking based on the analysis of video captured by a headworn camera. To facilitate our research, we first introduce the EGTEA Gaze+ dataset. Our dataset comes with videos, gaze tracking data, hand masks and action annotations, thereby providing the most comprehensive benchmark for First Person Vision (FPV). Moving beyond the dataset, we propose a novel deep model for joint gaze estimation and action recognition in FPV. Our method describes the participant's gaze as a probabilistic variable and models its distribution using stochastic units in a deep network. We further sample from these stochastic units, generating an attention map to guide the aggregation of visual features for action recognition. Our method is evaluated on our EGTEA Gaze+ dataset and achieves a performance level that exceeds the state-of-the-art by a significant margin. More importantly, we demonstrate that our model can be applied to larger scale FPV dataset-EPIC-Kitchens even without using gaze, offering new state-of-the-art results on FPV action recognition.

show abstract

Head Motion Signatures from Egocentric Videos

Cited by 25 publications

References 13 publications

Identifying First-Person Camera Wearers in Third-Person Videos

Identifying First-Person Camera Wearers in Third-Person Videos

Future Person Localization in First-Person Videos

In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video

Contact Info

Product

Resources

About