Tree Memory Networks for Modelling Long-term Temporal Dependencies

Kosaraju

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

et al. 2019

770

798

This paper addresses the problem of path prediction for multiple interacting agents in a scene, which is a crucial step for many autonomous platforms such as self-driving cars and social robots. We present SoPhie; an interpretable framework based on Generative Adversarial Network (GAN), which leverages two sources of information, the path history of all the agents in a scene, and the scene context information, using images of the scene. To predict a future path for an agent, both physical and social information must be leveraged. Previous work has not been successful to jointly model physical and social interactions. Our approach blends a social attention mechanism with a physical attention that helps the model to learn where to look in a large scene and extract the most salient parts of the image relevant to the path. Whereas, the social attention component aggregates information across the different agent interactions and extracts the most important trajectory information from the surrounding neighbors. SoPhie also takes advantage of GAN to generates more realistic samples and to capture the uncertain nature of the future paths by modeling its distribution. All these mechanisms enable our approach to predict socially and physically plausible paths for the agents and to achieve state-of-the-art performance on several different trajectory forecasting benchmarks.

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints

Kosaraju

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

et al. 2019

770

798

“…As we are dealing with longer sequences up to a duration of 5 minutes, the modelling ability of the LSTM is limited as LSTMs struggle to capture long-term dependencies when sequences are very long [3,10]. To address this limitation, we make use of memory networks [3,10] together with LSTM models, to improve the ability to capture long-term relationships.…”

Section: A1mentioning

confidence: 99%

“…However, none of the existing works in either action anticipation or future action sequence prediction have investigated how best to capture long-term dependencies. We speculate that LSTMs fail to capture such dependencies as they model only the relationships within a given sequence [3]. To address this, we demonstrate a method to effectively capture these relationships using neural memory networks.…”

Section: Related Workmentioning

confidence: 99%

“…Neural Memory Networks: When anticipating future actions it is essential to be equiped with a model that can map long-term relationships among the observed actions. Memory networks can facilitate this process and these have been widely used in different areas of computer vision [3,14] to model and respond to such long-term temporal relationships. However, in the area of action recognition only a limited number of methods are supported by memory networks [16,21].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Forecasting Future Action Sequences with Neural Memory Networks

Gammulle¹,

Denman²,

Sridharan³

et al. 2019

Preprint

Self Cite

We propose a novel neural memory network based framework for future action sequence forecasting. This is a challenging task where we have to consider short-term, within sequence relationships as well as relationships in between sequences, to understand how sequences of actions evolve over time. To capture these relationships effectively, we introduce neural memory networks to our modelling scheme. We show the significance of using two input streams, the observed frames and the corresponding action labels, which provide different information cues for our prediction task. Furthermore, through the proposed method we effectively map the long-term relationships among individual input sequences through separate memory modules, which enables better fusion of the salient features. Our method outperforms the state-of-the-art approaches by a large margin on two publicly available datasets: Breakfast and 50 Salads.

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Deep Decision Trees for Discriminative Dictionary Learning with Adversarial Multi-agent Trajectories

Fernando

Sridharan

Fookes

et al. 2018

Self Cite

With the explosion in the availability of spatio-temporal tracking data in modern sports, there is an enormous opportunity to better analyse, learn and predict important events in adversarial group environments. In this paper, we propose a deep decision tree architecture for discriminative dictionary learning from adversarial multi-agent trajectories. We first build up a hierarchy for the tree structure by adding each layer and performing feature weight based clustering in the forward pass. We then fine tune the player role weights using back propagation. The hierarchical architecture ensures the interpretability and the integrity of the group representation. The resulting architecture is a decision tree, with leaf-nodes capturing a dictionary of multiagent group interactions. Due to the ample volume of data available, we focus on soccer tracking data, although our approach can be used in any adversarial multi-agent domain. We present applications of proposed method for simulating soccer games as well as evaluating and quantifying team strategies.