Action Anticipation with RBF Kernelized Feature Mapping RNN

Abstract

We introduce a novel Recurrent Neural Network-based algorithm for future video feature generation and action anticipation called feature mapping RNN . Our novel RNN architecture builds upon three effective principles of machine learning, namely parameter sharing, Radial Basis Function kernels and adversarial training. Using only some of the earliest frames of a video, the feature mapping RNN is able to generate future features with a fraction of the parameters needed in traditional RNN. By feeding these future features into a simple multilayer perceptron facilitated with an RBF kernel layer, we are able to accurately predict the action in the video.In our experiments, we obtain 18% improvement on JHMDB-21 dataset, 6% on UCF101-24 and 13% improvement on UT-Interaction datasets over prior stateof-the-art for action anticipation.

Predicting the Future: A Jointly Learnt Model for Action Anticipation

“…In Tab. 2, the results for UCF101-24 shows that our model is able to outperform RBF-RNN [45] by 0.9% while in Tab. 3 we outperform [45] on the UTI dataset by 1.3% at the earliest setting.…”

“…The authors of RBF-RNN [45] use a GAN learning process where the loss function is also automatically learnt. Similar to the proposed architecture, the RBF-RNN [45] model also utilises the spatial representation of the scene through a Deep CNN model and tries to predict the future scene representations. However in contrast to the proposed architecture this method does not utilise temporal features, or joint learning.…”

“…Method Accuracy Temporal Fusion [11] 86.0 ROAD [47] 92.0 ROAD + BroxFlow [47] 90.0 RBF-RNN [45] 98.0 Proposed 98.9 Table 2. Action anticipation results for UCF101-24 considering 50% of frames from each video.…”

“…Upon training this representation, they train a classifier on the predicted features to anticipate the future action class. We argue that the approach of [45] is suboptimal, as there is no guarantee that the future action representation is well suited to predicting the action due to the two stage learning. Our approach, which learns the tasks jointly, ensures that a rich multi-modal embedding is learnt that captures the salient information needed for both tasks.…”

“…In contrast to recent works [3,45,50] which rely solely on visual inputs, and inspired by [10,17,31], we propose a joint learning process which attends to salient components of both visual and temporal streams, and builds a highly informative context descriptor for future action anticipation. In [50] the authors demonstrate that the context semantics, which capture high level action related concepts including environmental details, objects, and historical actions and interactions, are more important when anticipating actions than the actual pixel values of future frames.…”