Deep RNN Framework for Visual Sequential Applications

Abstract

Extracting temporal and representation features efficiently plays a pivotal role in understanding visual sequence information. To deal with this, we propose a new recurrent neural framework that can be stacked deep effectively. There are mainly two novel designs in our deep RNN framework: one is a new RNN module called Context Bridge Module (CBM) which splits the information flowing along the sequence (temporal direction) and along depth (spatial representation direction), making it easier to train when building deep by balancing these two directions; the other is the Overlap Coherence Training Scheme that reduces the training complexity for long visual sequential tasks on account of the limitation of computing resources.We provide empirical evidence to show that our deep RNN framework is easy to optimize and can gain accuracy from the increased depth on several visual sequence problems. On these tasks, we evaluate our deep RNN framework with 15 layers, 7× than conventional RNN networks, but it is still easy to train. Our deep framework achieves more than 11% relative improvements over shallow RNN models on Kinetics, UCF-101, and HMDB-51 for video classification. For auxiliary annotation, after replacing the shallow RNN part of Polygon-RNN with our 15-layer deep CBM, the performance improves by 14.7%. For video future prediction, our deep RNN improves the state-of-the-art shallow model's performance by 2.4% on PSNR and SSIM. The code and trained models are published accompanied by this paper: https://github. com/BoPang1996/Deep-RNN-Framework.

Transferable Interactiveness Knowledge for Human-Object Interaction Detection

“…As a sub-task of visual relationship [16,19], HOI is strongly related to the human body and object understanding [32,34,35,22,36,40,33]. It is crucial for behavior understanding and can facilitate activity understanding [2,37], imitation learning [1], etc. Recently, impressive progress has been made by utilizing Deep Neural Networks (DNNs) in this area [3,12,21,9].…”

A Survey on Heterogeneous One-class Collaborative Filtering

“…With the great success of integration with deep learning in various domains such as natural language processing [Kim 2014;Li 2017] and computer vision Pang et al 2019], recommender systems are also refreshed unprecedentedly by deep learning techniques, in which the sub-domains include homogeneous/heterogeneous one-class collaborative filtering [Gao et al 2020;, sequential recommendation [Jannach and Ludewig 2017;Tang and Wang 2018], click-through rate prediction [Cheng et al 2016;Guo et al 2017b], and so forth. In this section, we will focus on deep learning-based recommendation methods via different neural architectures, i.e., multi-layer perceptron, autoencoder, memory networks, recurrent neural network, and reinforcement learning, for solving OCCF and HOCCF problems.…”

Complex sequential understanding through the awareness of spatial and temporal concepts

“…These tricks are not the focus of this paper. Compared with CBM [30], the new SCS with sub-tasks and ASTSGD achieves better performances. [4]).…”

“…The Deep RNN framework [30] is the predecessor to the SCS described in this work, yet they have significant differences. Firstly, in the Deep RNN framework, the splitting of two flows is designed to make the deep recurrent structure easier to train by adding spatial shortcuts over temporal flows.…”