Scale coding bag of deep features for human attribute and action recognition

Abstract

Most approaches to human attribute and action recognition in still images are based on image representation in which multi-scale local features are pooled across scale into a single, scale-invariant encoding. Both in bagof-words and the recently popular representations based on convolutional neural networks, local features are computed at multiple scales. However, these multi-scale convolutional features are pooled into a single scale-invariant representation. We argue that entirely scale-invariant image representations are sub-optimal and investigate approaches to scale coding within a bag of deep features framework. Our approach encodes multi-scale information explicitly during the image encoding stage. We propose two strategies to encode multi-scale information explicitly in the final image representation. We validate our two scale coding techniques on five datasets: Willow, PASCAL VOC 2010, PASCAL VOC 2012, Stanford-40 and Human Attributes (HAT-27). On all datasets, the proposed scale coding approaches outperform both the scale-invariant method and the standard deep features of the same network. Further, combining our scale coding approaches with standard deep features leads to consistent improvement over the state of the art.

Action Recognition in Still Images Based on R-FCN Detector

“…Yan, Smith and Zhang (2017) encode the CNN image patches features generated by a region proposal algorithm with VLAD(Vector of Locally Aggregated Descriptors) and subsequently represent an image by a compact code, which the authors claim that it captures more fine-grained properties of the images and also contains global contextual information. Khan et al (2018) investigate approaches to scale coding within a bag of deep features framework, demonstrating that both absolute and relative scale information can be encoded in final image representations and that relaxing the traditional scale invariance commonly employed in image classification can lead to significant gains in recognition performance, achieving state-of-the-art-results. Nevertheless, most of these approaches concentrate on the classification task, disregarding the detection task, while real-world images often have multiple actions being performed, where action detection plays a significant role.…”

“…Our evaluation results are obtained on the testing images through the "boxless action classification" of the publicly available competition server and the mAP is computed for evaluation. Table 2 presents the comparison results with the state-of-the-art approaches (Gkioxari et al (2015), Qi et al (2017) and Khan et al (2018)) on the VOC Action 2012 test set. As shown, general person detectors perform worse in this dataset, we suspect that this occurs due to the fact that in this dataset humans interact with larger objects, such as bicycles and horses, and this lack of information proved to be prejudicial to recognition, sustaining our hypothesis that the human-object interaction is fundamental for action recognition.…”

“…We highlight that our approach yields a significant gain of 3.7% in mAP when compared to the stateof-the-art. Method mAP Gkioxari et al (2015) 90.2% Qi et al (2017) 89.8% Khan et al (2018) 83.9% GP 64.2% GA 93.9% SA 80.6%…”

Two-Stream Part-Based Deep Representation for Human Attribute Recognition

“…In the scale-coded image representation, feature scales are divided into several scale subgroups S t that partition the whole set of extracted scales (i.e. [21]. The multi-scale convolutional features are then pooled using Fisher vector (FV) encoding scheme.…”

“…As discussed earlier, deep features extracted from activations of the fully connected (FC) layers of the deep CNNs are typically used for image representation. Instead of FC layers, activations from the last convolutional layer of the deep networks have been shown to provide excellent performance in recent works [19], [20], [21]. The convolutional layers are known to be discriminative and semantically meaningful.…”