A Bag of Expression framework for improved human action recognition

Abstract

The Bag of Words (BoW) approach has been widely used for human action recognition in recent state-of-the-art methods. In this paper, we introduce what we call a Bag of Expression (BoE) framework, based on the bag of words method, for recognizing human action in simple and realistic scenarios. The proposed approach includes space time neighborhood information in addition to visual words. The main focus is to enhance the existing strengths of the BoW approach like view independence, scale invariance and occlusion handling. BOE includes independent pairs of neighbors for building expressions, therefore it is tolerant to occlusion and capable of handling view independence up to some extent in realistic scenarios. Our main contribution includes learning a class specific visual words extraction approach for establishing a relationship between these extracted visual words in both space and time dimension. Finally, we have carried out a set of experiments to optimize different parameters and compare its performance with recent state-of-the-art-methods. Our approach outperforms existing Bag of Words based approaches, when evaluated using the same performance evaluation methods. We tested our approach on four publicly available datasets for human action recognition i.e. UCF-Sports, KTH, UCF11 and UCF50 and achieve significant results i.e. 97.3%, 99.5%, 96.7% and 93.42% respectively in terms of average accuracy.

Instructor Activity Recognition through Deep Spatiotemporal Features and Feedforward Extreme Learning Machines

“…The C3D features are computed from RGB IAVID-1 videos and produce 48.77% and 40% prediction accuracy using SVM and CNN. The performance of C3D features is comparable to 2D CNN without considering temporal information for HAR at frame level [18] Similarly, Bag of Expression [32] for HAR produces 26.67% recognition rate using handcrafted 3D-Harris and 3D-SIFT. Some recent silhouettes based HAR techniques are using MHIs described through HOG [20] and LBP-HOG [31] descriptor to recognize human activities through nearest neighbor and SVM classifiers.…”

“…The proposed technique outperforms other techniques based on silhouettes in terms of precise recognition. To this end, we analyzed and compared our technique with methods [17,20,31,32] on the IAVID-I dataset. The C3D features are computed from RGB IAVID-1 videos and produce 48.77% and 40% prediction accuracy using SVM and CNN.…”

“…HOG and LBP representation of MHI with SVM classifier [31] 55% Harris 3D and HOG 3D with BOE [32] 26.67% Harris 3D, HOG/ HOF, BoF with MCV-ELM [29] 13.33% Harris 3D, HOG/ HOF, BoF with MV-ELM [30] 13.33%…”

Vehicle Make and Model Recognition using Bag of Expressions

“…Two different techniques have been proposed for dictionary building in [25]: modular dictionary and single dictionary. In [26,27] Nazir et al proposed the dynamic spatio-temporal bag of expressions (D-STBoE) model and the BoE framework for action recognition which improves the existing strength of bag of words. A global feature ensemble representation is discussed by Chen et al [18] who combined the HOG vehicle features extracted in a grid-based pattern.…”

Dynamic Spatio-Temporal Bag of Expressions (D-STBoE) Model for Human Action Recognition

“…However, since BoVW approaches only consider the frequency of each visual word and ignore the spatio-temporal relation among visual words, such representation is not able to exploit the contextual relationship between visual words. To overcome this drawback, we propose an extension of our previously proposed Bag of Expressions (BoE) model [7], which incorporates contextual relationships of visual words while preserving inherent qualities of the classical BoVW approach. The main idea of visual expression formation is to store the spatio-temporal contextual information that is usually lost during formation of visual words by encoding neighboring spatio-temporal interest points’ information.…”