Making Action Recognition Robust to Occlusions and Viewpoint Changes

Abstract

Most state-of-the-art approaches to action recognition rely on global representations either by concatenating local information in a long descriptor vector or by computing a single location independent histogram. This limits their performance in presence of occlusions and when running on multiple viewpoints. We propose a novel approach to providing robustness to both occlusions and viewpoint changes that yields significant improvements over existing techniques. At its heart is a local partitioning and hierarchical classification of the 3D Histogram of Oriented Gradients (HOG) descriptor to represent sequences of images that have been concatenated into a data volume. We achieve robustness to occlusions and viewpoint changes by combining training data from all viewpoints to train classifiers that estimate action labels independently over sets of HOG blocks. A top level classifier combines these local labels into a global action class decision.

Fusion of Skeletal and Silhouette-Based Features for Human Action Recognition with RGB-D Devices

“…A bag of rectangles has been proposed in [9], where a histogram of oriented rectangular patches is extracted over the whole silhouette. Similarly, in [23], a 3D histogram of oriented gradients (3DHOG) is extracted on densely distributed regions. A very popular feature is the one from Tran and Sorokin [20].…”

Should we discard sparse or incomplete videos?

“…(2) The second type is caused by partial occlusion lasting throughout all frames. The work in [7] explored this type by artificially placing occluders on some predefined parts in the human body. This method however needs to Occluded frames under different occlusion settings for the KTH running action.…”

“…when using a compressive sensing device). Some approaches have also been proposed to handle occlusions in tracking or detection [5][6] [7], but they may still fail under severe occlusions.…”

Thresholding a Random Forest Classifier

“…Bernard et al propose a Dynamic [4]. Example images of the human action recognition datasets: (c) IXMAS multi-view [5,6] (d) KTH [7] (e) Weizman [8,9].…”

“…We manually labeled the dataset and used the bounding boxes for the classification. The bounding boxes are available for download at our homepage 5 . IXMAS: Additionally, our framework is evaluated on the IXMAS dataset for multiview action recognition [5,6].…”