Magnitude-Orientation Stream network and depth information applied to activity recognition

Abstract

The temporal component of videos provides an important clue for activity recognition, as a number of activities can be reliably recognized based on the motion information. In view of that, this work proposes a novel temporal stream for two-stream convolutional networks based on images computed from the optical flow magnitude and orientation, named Magnitude-Orientation Stream (MOS), to learn the motion in a better and richer manner. Our method applies simple non-linear transformations on the vertical and horizontal components of the optical flow to generate input images for the temporal stream. Moreover, we also employ depth information to use as a weighting scheme on the magnitude information to compensate the distance of the subjects performing the activity to the camera. Experimental results, carried on two well-known datasets (UCF101 and NTU), demonstrate that using our proposed temporal stream as input to existing neural network architectures can improve their performance for activity recognition. Results demonstrate that our temporal stream provides complementary information able to improve the classical two-stream methods, indicating the suitability of our approach to be used as a temporal video representation.

Motion-Based Representations For Activity Recognition

“…Contributions of this work correspond to the main achievements during the doctorate research. Methods were published on the International Conference on Pattern Recognition (ICPR) 2016 [34] (awarded with an IAPR Travel Stipend), Conference on Graphics, Patterns and Images (SIBGRAPI) 2017 [35] (awarded as the best Computer Vision/Image Processing/Pattern Recognition main track paper award), International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISAPP) 2018 [36], IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) 2019 [37], Conference on Graphics, Patterns and Images (SIBGRAPI) 2019 [26] and a publication in the Journal of Visual Communication and Image Representation (JVCI) 2019 [38].…”