First-Person Action Decomposition and Zero-Shot Learning

Knowledge guided learning: Open world egocentric action recognition with zero supervision

“…Zero-shot learning models (Zhang, Li, and Rehg 2017;Jain et al 2015;Liu, Kuipers, and Savarese 2011) do not require as much supervision and learn semantic correspondences that extend beyond training classes to unseen test classes. The common approaches are to either use an attribute space or embedding space that captures the semantics of a scene and helps extend beyond the training label by exploiting the semantic correspondences across classes.…”

“…GTEA Gaze contains 10 different verbs and 38 different nouns, while GTEA Gaze+ contains 15 verbs and 27 nouns. We report results averaged over all subjects for a fair comparison with prior works (Ma, Fan, and Kitani 2016;Zhang, Li, and Rehg 2017), which use leave-one-out cross-validation. We also test our approach's generalization capability to scenes beyond egocentric videos for object detection with zero supervision.…”

“…We use a two-stream CNN (Ma, Fan, and Kitani 2016) as our fully supervised baseline, given its tremendous success in action recognition. We also establish a zeroshot baseline based (Zhang, Li, and Rehg 2017) for comparison on the object recognition and activity recognition tasks.…”

“…Advances in deep learning have enabled the development of models that have exhibited a remarkable tendency to recognize (Fathi, Li, and Rehg 2012;Liu, Kuipers, and Savarese 2011;Singh, Arora, and Jawahar 2016;Sudhakaran, Escalera, and Lanz 2019;Zhang, Li, and Rehg 2017) and localize actions (Aakur and Sarkar 2020;Jain et al 2015) in videos. However, they tend to experience errors when faced with scenes or examples beyond their initial training environment.…”

Learning to Localize and Align Fine-Grained Actions to Sparse Instructions

“…Also the recipes are not treated as a strictly ordered set since recipe steps can be done out of order. First Person Vision: Our system is created for first person (FP) videos which have become more prevalent in the computer vision community in recent years [18,16,22,13,33]. We utilize the egocentric cues proposed by [16] in our method for action proposal generation.…”