Interpretable Spatio-Temporal Attention for Video Action Recognition

Abstract

arXiv:1810.04511v2 [cs.CV] 3 Jun 2019 frame 1 frame 2 frame n CNN CNN CNN Spatial Attention Spatial Attention Spatial Attention Temporal Attention Temporal Attention Temporal Attention Convolution LSTM Convolution LSTM Convolution LSTM AVG Playing Volleyball X1 < l a t e x i t s h a 1 _ b a s e 6 4 = " B G + 8 X 3 r Z G o V 3 v z 6 / F F z d q k I g 7 b M = " > A A A B 6 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f Q k B S 8 e K 9 o P a E P Z b D f t 0 s 0 m 7 E 6 E E v o T v H h Q x K u / y J v / x m 2 a g 7 a + s P D w z g w 7 8 w a J F A Z d 9 9 s p r a 1 v b G 6 V t y s 7 u 3 v 7 B 9 X D o 7 a J U 8 1 4 i 8 U y 1 t 2 A G i 6 F 4 i 0 U K H k 3 0 Z x G g e S d Y H I 7 r 3 e e u D Y i V o 8 4 T b g f 0 Z E S o W A U r f X Q H X i D a s 2 t u 7 n I K n g F 1 K B Q c 1 D 9 6 g 9 j l k Z c I Z P U m J 7 n J u h n V K N g k s 8 q / d T w h L I J H f G e R U U j b v w s X 3 V G z q w z J G G s 7 V N I c v f 3 R E Y j Y 6 Z R Y D s j i m O z X J u b / 9 V 6 K Y b X f i Z U k i J X b P F R m E q C M Z n f T Y Z C c 4 Z y a o E y L e y u h I 2 p p g x t O h U b g r d 8 8 i q 0 L + q e 5 f v L W u O m i K M M J 3 A K 5 + D B F T T g D p r Q A g Y j e I Z X e H O k 8 + K 8 O x + L 1 p J T z B z D H z m f P 9 m v j X w = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " B G + 8 X 3 r Z G o V 3 v z 6 / F F z d q k I g 7 b M = " > A A A B 6 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f Q k B S 8 e K 9 o P a E P Z b D f t 0 s 0 m 7 E 6 E E v o T v H h Q x K u / y J v / x m 2 a g 7 a + s P D w z g w 7 8 w a J F A Z d 9 9 s p r a 1 v b G 6 V t y s 7 u 3 v 7 B 9 X D o 7 a J U 8 1 4 i 8 U y 1 t 2 A G i 6 F 4 i 0 U K H k 3 0 Z x G g e S d Y H I 7 r 3 e e u D Y i V o 8 4 T b g f 0 Z E S o W A U r f X Q H X i D a s 2 t u 7 n I K n g F 1 K B Q c 1 D 9 6 g 9 j l k Z c I Z P U m J 7 n J u h n V K N g k s 8 q / d T w hc 4 Z y a o E y L e y u h I 2 p p g x t O h U b g r d 8 8 i q 0 L + q e 5 f v L W u O m i K M M J 3 A K 5 + D B F T T g D p r Q A g Y j e I Z X e H O k 8 + K 8 O x + L 1 p J T z B z D H z m f P 9 m v j X w = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " B G + 8 X 3 r Z G o V 3 v z 6 / F F z d q k I g 7 b M = " > A A A B 6 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f Q k B S 8 e K 9 o P a E P Z b D f t 0 s 0 m 7 E 6 E E v o T v H h Q x K u / y J v / x m 2 a g 7 a + s P D w z g w 7 8 w a J F A Z d 9 9 s p r a 1 v b G 6 V t y s 7 u 3 v 7 B 9 X D o 7 a J U 8 1 4 i 8 U y 1 t 2 A G i 6 F 4 i 0 U K H k 3 0 Z x G g e S d Y H I 7 r 3 e e u D Y i V o 8 4 T b g f 0 Z E S o W A U r f X Q H X i D a s 2 t u 7 n I K n g F 1 K B Q c 1 D 9 6 g 9 j l k Z c I Z P U m J 7 n J u h n V K N g k s 8 q / d T w h L I J H f G e R U U j b v w s X 3 V G z q w z J G G s 7 V N I c v f 3 R E Y j Y 6 Z R Y D s j i m O z X J u b / 9 V 6 K Y b X f i Z U k i J X b P F R m E q C M Z n f T Y Z C c 4 Z y a o E y L e y u h I 2 p p g x t O h U b g r d 8 8 i q 0 L + q e 5 f v L W u O m i K M M J 3 A K 5 + D B F T T g D p r Q A g Y j e I Z X e H O k 8 + K 8 O x + L 1 p J T z B...

Recognizing Human-Object Interaction in Multi-Camera Environments

“…A soft attention mechanism [1] was inserted before the feature extraction model and is responsible for blacking-out irrelevant parts of input to minimize their impacts on prediction. The spatial attention block consists of a convolutional network with the 'same' padding in all layers that learns to produce an importance mask for each input image, which is then multiplied element-wise with the original input image.…”

Human Action Recognition in Unconstrained Trimmed Videos Using Residual Attention Network and Joints Path Signature

“…In [27], the authors mention a hierarchical attention network for document classification. Recently, attention network has also been proposed for action recognition in videos [28], [29], [30]. In [28], the authors introduce an attention mechanism as low-rank second-order pooling for single image classification.…”

“…In [29], the authors developed an attention-based neural network in order to model the scene objects interaction for action recognition and video captioning. In [30], authors introduce two separate temporal and spatial attention mechanism to identify the most relevant frame and the most-relevant spatial location in that frame of a particular action.…”

Spatio-Temporal Attention Deep Recurrent Q-Network for POMDPs

“…Temporal attention performs the same selective approach as seen in spatial attention but in the time domain, choosing certain timesteps over others. Although there is a smaller body of research focusing on temporal attention in the machine learning community [25,17,11] , it is well-established in the neuroscientific community that the combination of spatial and temporal attention in human perception is a critical aspect of perceiving occulted and moving objects in time [3] [10], two for the most prominent features of a flickering Atari game screen. The authors combine spatial and temporal attention in an attempt to recreate the additive effects [10] seen in human perception, in a deep RL agent.…”