Learning Action Maps of Large Environments via First-Person Vision

Abstract

When people observe and interact with physical spaces, they are able to associate functionality to regions in the environment. Our goal is to automate dense functional understanding of large spaces by leveraging sparse activity demonstrations recorded from an ego-centric viewpoint. The method we describe enables functionality estimation in large scenes where people have behaved, as well as novel scenes where no behaviors are observed. Our method learns and predicts "Action Maps", which encode the ability for a user to perform activities at various locations. With the usage of an egocentric camera to observe human activities, our method scales with the size of the scene without the need for mounting multiple static surveillance cameras and is well-suited to the task of observing activities up-close. We demonstrate that by capturing appearance-based attributes of the environment and associating these attributes with activity demonstrations, our proposed mathematical framework allows for the prediction of Action Maps in new environments. Additionally, we offer a preliminary glance of the applicability of Action Maps by demonstrating a proof-ofconcept application in which they are used in concert with activity detections to perform localization.

A Survey on Deep Learning Based Approaches for Action and Gesture Recognition in Image Sequences

“…Immediate effort is also expected in action/gesture localization in long, untrimmed, and realistic videos [128,34,95]. As such, we envision newer problems like early recognition [28], multi-task learning [127], captioning, recognition from low resolution sequences [66] and lifelog devices [87] will receive attention in the next years.…”

ASTRON: Action-Based Spatio-Temporal Robot Navigation

“…Environmental representations that describe both the data related to the object and to the action in the environment have been proposed. Action maps have been proposed as environment representations focusing on actions, which embed the action possibility in real space based on the history of human activities [14], [15]. In addition, another approach is to apply an object classification method based on the concept of affordance to associate the actions with objects [16], [17].…”

Deep Learning for Action and Gesture Recognition in Image Sequences: A Survey

“…Additional effort is expected to advance in the research of methods able to simultaneously perform both detection and recognition tasks in long, realistic videos (Gkioxari and Malik, 2015;Shou et al, 2016b). As such, we envision other related problems like early recognition Escalante et al (2016a), multi task learning , captioning, recognition from low resolution sequences Nasrollahi et al (2015) and from lifelog devices Rhinehart and Kitani (2016) will receive special attention within the next few years.…”