Where to Focus on for Human Action Recognition?

Abstract

In this paper, we present a new attention model for the recognition of human action from RGB-D videos. We propose an attention mechanism based on 3D articulated pose. The objective is to focus on the most relevant body parts involved in the action. For action classification, we propose a classification network compounded of spatio-temporal subnetworks modeling the appearance of human body parts and RNN attention subnetwork implementing our attention mechanism. Furthermore, we train our proposed network end-to-end using a regularized cross-entropy loss, leading to a joint training of the RNN delivering attention globally to the whole set of spatio-temporal features, extracted from 3D ConvNets. Our method outperforms the State-ofthe-art methods on the largest human activity recognition dataset available to-date (NTU RGB+D Dataset) which is also multi-views and on a human action recognition dataset with object interaction (Northwestern-UCLA Multiview Action 3D Dataset).

Looking deeper into Time for Activities of Daily Living Recognition

“…The attention mechanism of non-local blocks [35] from convolutional feature maps are not viewinvariant and thus perform worse than simple I3D as backbone of the Temporal Model in CV protocols. P-I3D [8] with 42M trainable parameters as compared to simple I3D's 12M trainable parameters outperforms the state-of-the-art results on NTU (95% average over CS and CV) and NU-CLA (93.5%) datasets when used as a backbone of the Temporal Model. The Global Model with P-I3D as base network has 80M trainable parameters and improves action, with similar motion like wearing glasses (+2.5%) and taking off glasses (+2.1%) compared to the Basic Model (P-I3D).…”

“…However, this operation computing the affinity between the features does not go beyond the spatio-temporal cube, thus does not account for long-term temporal relations. For ADL recognition, Das et al [8] proposed a spatial attention mechanism on the spatio-temporal features extracted from I3D network. The spatial attention provides soft-weights to the pertinent human body parts relevant to the action.…”

“…Then we propose a focus of attention on -the pertinent temporal segment in a video and the pertinent temporal granularity. Furthermore, the capability of the Temporal Model to combine with the existing 3D CNNs [8,35] stands it out from the existing approaches [34,37] for temporally complex actions.…”

“…Consequently, classification is not affected by the wrong poses. Table 2, 3 also shows the effectiveness of our Temporal Model when adopted on top of rich discriminative features from existing spatio-temporal attention models [8,35]. We call them Global Model (I3D-NL base) or (P-I3D base) -the base network in parentheses.…”

“…We make two hypotheses: the input video clip contains a single class label, and the articulated poses are available. Inspired from the recent trend of using poses to guide RGB cue [3,4,8], we take the articulated poses as input to the attention module. The articulated poses are highly informative, robust to rotation and illumination, and thus provides a strong clue to select the pertinent subsequences in a video.…”