Regional Attention Network (RAN) for Head Pose and Fine-Grained Gesture Recognition

Abstract

Affect is often expressed via non-verbal body language such as actions/gestures, which are vital indicators for human behaviors. Recent studies on recognition of fine-grained actions/gestures in monocular images have mainly focused on modeling spatial configuration of body parts representing body pose, human-objects interactions and variations in local appearance. The results show that this is a brittle approach since it relies on the accurate body parts/objects detection. In this work, we argue that there exist local discriminative semantic regions, whose "informativeness" can be evaluated by the attention mechanism for inferring fine-grained gestures/actions. To this end, we propose a novel end-to-end Regional Attention Network (RAN), which is a fully Convolutional Neural Network (CNN) to combine multiple contextual regions through attention mechanism, focusing on parts of the images that are most relevant to a given task. Our regions consist of one or more consecutive cells and are adapted from the strategies used in computing HOG (Histogram of Oriented Gradient) descriptor. The model is extensively evaluated on ten datasets belonging to 3 different scenarios: 1) head pose recognition, 2) drivers state recognition, and 3) human action and facial expression recognition. The proposed approach outperforms the state-of-the-art by a considerable margin in different metrics.

An attention-driven hierarchical multi-scale representation for visual recognition

“…This is mainly because the parameters are shared between regions belonging to a given graph. Our model is also comparable to CAP [4] (34.2M) and lighter than RAN [3] (49M). Our per-image inference time is 8.5 milliseconds (ms).…”

“…Our hierarchical multi-scale regions approach is motivated by recent region-based approaches [3,4,22,25,69,82] in solving FGVC. However, it is different since we use smaller (look closer) to larger (look from far) regions in a hierarchical fashion.…”

“…However, it is different since we use smaller (look closer) to larger (look from far) regions in a hierarchical fashion. Whereas, the regions in [22,25,69,82] are considered as object proposals from a detector and the approaches in [3,4] use multi-scale regions by exploring HOG cells and blocks. Our regions within a layer in the hierarchy have the same area but with different aspect ratios (Fig.…”

“…Pre-trained ImageNet weights are used to initialize Xception for quicker convergence [24]. Regions of the fixed area but with different aspect ratios corresponding to a given hierarchical layer are generated using the region proposal algorithm in [3]. A single-layer GCN (Fig.…”

“…For a 3-layer hierarchical structure (Fig. 1b), there are 52 possible regions (layer 1: 25, layer 2: 22, and layer 3: 5) generated using the region proposal in [3]. Input images are resized to 256×256, and data augmentation of random rotation and random zoom are used.…”