Social Behavior Prediction from First Person Videos

Su, Shan; Hong, Jung Pyo; Shi, Jianbo; Park, Hyun Soo

doi:10.48550/arxiv.1611.09464

Cited by 1 publication

(1 citation statement)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other trajectory forecasting approaches use demonstrations observed from a bird'seye view; [31] infers latent goal locations and [2] employ LSTMs to jointly reason about trajectories of multiple humans. In [25], the model forecasted short-term future trajectories of a first-person camera wearer by retrieving the nearest neighbors from a dataset of first-person trajectories under an obstacle-avoidance cost function, with each trajectory representing predictions of where the user will move in view of the frame; in [26], a similar model with learned cost function is extended to multiple users. Predicting Future Behavior: In [6,21], the tasks are to recognize an unfinished event or activity.…”

Section: Related Workmentioning

confidence: 99%

First-Person Activity Forecasting with Online Inverse Reinforcement Learning

Rhinehart

Kitani

2017

2017 IEEE International Conference on Computer Vision (ICCV)

118

View full text Add to dashboard Cite

We address the problem of incrementally modeling and forecasting long-term goals of a first-person camera wearer: what the user will do, where they will go, and what goal they seek. In contrast to prior work in trajectory forecasting, our algorithm, DARKO, goes further to reason about semantic states (will I pick up an object?), and future goal states that are far in terms of both space and time. DARKO learns and forecasts from first-person visual observations of the user's daily behaviors via an Online Inverse Reinforcement Learning (IRL) approach. Classical IRL discovers only the rewards in a batch setting, whereas DARKO discovers the states, transitions, rewards, and goals of a user from streaming data. Among other results, we show DARKO forecasts goals better than competing methods in both noisy and ideal settings, and our approach is theoretically and empirically no-regret.

show abstract