Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos

Abstract

We introduce an approach for pre-training egocentric video models using large-scale third-person video datasets. Learning from purely egocentric data is limited by low dataset scale and diversity, while using purely exocentric (third-person) data introduces a large domain mismatch. Our idea is to discover latent signals in third-person video that are predictive of key egocentric-specific properties. Incorporating these signals as knowledge distillation losses during pre-training results in models that benefit from both the scale and diversity of third-person video data, as well as representations that capture salient egocentric properties. Our experiments show that our "Ego-Exo" framework can be seamlessly integrated into standard video models; it outperforms all baselines when fine-tuned for egocentric activity recognition, achieving state-of-the-art results on Charades-Ego and EPIC-Kitchens-100.

Learning Visual Affordance Grounding from Demonstration Videos

“…4) Transferability: Although third-person videos are more accessible, in cases such as robot manipulation where the agent observes the environment and objects from the firstperson perspective, it is necessary to transfer the knowledge from third-person videos to first-person scenarios. However, transferring affordance grounding knowledge between different perspectives [78] is still under-explored and of practical meaning.…”