Non-Local Recurrent Neural Memory for Supervised Sequence Modeling

Fu, Canmiao; Pei, Wenjie; Cao, Qiong; Zhang, Chaopeng; Zhao, Yong; Shen, Xiaoyong; Tai, Yu-Wing

doi:10.1109/iccv.2019.00641

Cited by 10 publications

(6 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Inspired by non-local methods (Wang et al 2018;Fu et al 2019), we design a non-local recurrent regularization network to model long-range dependencies along the depth dimension, which helps to regularize cost volumes by aggregating more context information. As illustrated in Fig.…”

Section: Non-local Recurrent Regularizationmentioning

confidence: 99%

Non-local Recurrent Regularization Networks for Multi-view Stereo

Xu,

Oswald,

Tao

et al. 2021

Preprint

View full text Add to dashboard Cite

In deep multi-view stereo networks, cost regularization is crucial to achieve accurate depth estimation. Since 3D cost volume filtering is usually memory-consuming, recurrent 2D cost map regularization has recently become popular and has shown great potential in reconstructing 3D models of different scales. However, existing recurrent methods only model the local dependencies in the depth domain, which greatly limits the capability of capturing the global scene context along the depth dimension. To tackle this limitation, we propose a novel non-local recurrent regularization network for multi-view stereo, named NR2-Net. Specifically, we design a depth attention module to capture non-local depth interactions within a sliding depth block. Then, the global scene context between different blocks is modeled in a gated recurrent manner. This way, the long-range dependencies along the depth dimension are captured to facilitate the cost regularization. Moreover, we design a dynamic depth map fusion strategy to improve the algorithm robustness. Our method achieves state-of-the-art reconstruction results on both DTU and Tanks and Temples datasets.

show abstract

Section: Non-local Recurrent Regularizationmentioning

confidence: 99%

Non-local Recurrent Regularization Networks for Multi-view Stereo

Xu,

Oswald,

Tao

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Non-local Neural Networks [22] present non-local operations as a generic family of building blocks for capturing longrange temporal dependencies. Meanwhile, to capture highorder interactions between non-adjacent time steps, Non-local Recurrent Neural Memory [23] performs non-local operations to learn full-order interactions within a sliding temporal block and models global interactions between blocks in a gated recurrent manner. In addition, Zhou et al [24] propose Temporal Relation Network (TRN) to give CNNs a remarkable capability to discover temporal relations in video.…”

Section: Temporal Relation Modeling In Action Recognitionmentioning

confidence: 99%

Graph-Temporal LSTM Networks for Skeleton-Based Action Recognition

Zhu

Zhang

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Graph Convolutional Networks (GCNs), which model skeleton data as graphs, have obtained remarkable performance for skeleton-based action recognition. Particularly, the temporal dynamic of skeleton sequence conveys significant information in the recognition task. For temporal dynamic modeling, GCN-based methods only stack multi-layer 1D local convolutions to extract temporal relations between adjacent time steps. With the repeat of a lot of local convolutions, the key temporal information with non-adjacent temporal distance may be ignored due to the information dilution. Therefore, these methods still remain unclear how to fully explore temporal dynamic of skeleton sequence. In this paper, we propose a Temporal Enhanced Graph Convolutional Network (TE-GCN) to tackle this limitation. The proposed TE-GCN constructs temporal relation graph to capture complex temporal dynamic. Specifically, the constructed temporal relation graph explicitly builds connections between semantically related temporal features to model temporal relations between both adjacent and non-adjacent time steps. Meanwhile, to further explore the sufficient temporal dynamic, multi-head mechanism is designed to investigate multi-kinds of temporal relations. Extensive experiments are performed on two widely used largescale datasets, NTU-60 RGB+D and NTU-120 RGB+D. And experimental results show that the proposed model achieves the state-of-the-art performance by making contribution to temporal modeling for action recognition.

show abstract

“…An earlier conference version of this paper appeared in (Fu et al, 2019a). Compared to the prior version, this longer article is improved in three aspects.…”

Section: Introductionmentioning

confidence: 99%

Learning Sequence Representations by Non-local Recurrent Neural Memory

Pei

Feng

et al. 2022

Int J Comput Vis

Self Cite

View full text Add to dashboard Cite

The key challenge of sequence representation learning is to capture the long-range temporal dependencies. Typical methods for supervised sequence representation learning are built upon recurrent neural networks to capture temporal dependencies. One potential limitation of these methods is that they only model one-order information interactions explicitly between adjacent time steps in a sequence, hence the high-order interactions between nonadjacent time steps are not fully exploited. It greatly limits the capability of modeling the long-range temporal dependencies since the temporal features learned by one-order interactions cannot be maintained for a long term due to temporal information dilution and gradient vanishing. To tackle this limitation, we propose the Non-local Recurrent Neural Memory (NRNM) for supervised sequence representation learning, which performs non-local operations by means of self-attention mechanism to learn full-order interactions within a sliding temporal memory block and models global interactions between memory blocks in a gated recurrent manner. Consequently, our model is able to capture long-range dependencies. Besides, the latent highlevel features contained in high-order interactions can be distilled by our model. We validate the effectiveness and generalization of our NRNM on three types of sequence applications across different modalities, including sequence classification, step-wise sequential prediction and sequence similarity learning. Our model compares favorably against other state-of-the-art methods specifically designed for each of these sequence applications.

show abstract

Non-Local Recurrent Neural Memory for Supervised Sequence Modeling

Cited by 10 publications

References 30 publications

Non-local Recurrent Regularization Networks for Multi-view Stereo

Non-local Recurrent Regularization Networks for Multi-view Stereo

Graph-Temporal LSTM Networks for Skeleton-Based Action Recognition

Learning Sequence Representations by Non-local Recurrent Neural Memory

Contact Info

Product

Resources

About