Benchmarking a Multimodal and Multiview and Interactive Dataset for Human Action Recognition

Abstract

Human action recognition is an active research area in both computer vision and machine learning communities. In the past decades, the machine learning problem has evolved from conventional single-view learning problem, to cross-view learning, cross-domain learning and multitask learning, where a large number of algorithms have been proposed in the literature. Despite having large number of action recognition datasets, most of them are designed for a subset of the four learning problems, where the comparisons between algorithms can further limited by variances within datasets, experimental configurations, and other factors. To the best of our knowledge, there exists no dataset that allows concurrent analysis on the four learning problems. In this paper, we introduce a novel multimodal and multiview and interactive (M2I) dataset, which is designed for the evaluation of human action recognition methods under all four scenarios. This dataset consists of 1760 action samples from 22 action categories, including nine person-person interactive actions and 13 person-object interactive actions. We systematically benchmark state-of-the-art approaches on M2I dataset on all four learning problems. Overall, we evaluated 13 approaches with nine popular feature and descriptor combinations. Our comprehensive analysis demonstrates that M2I dataset is challenging due to significant intraclass and view variations, and multiple similar action categories, as well as provides solid foundation for the evaluation of existing state-of-the-art algorithms.

Improving Interpretability of Deep Neural Networks with Semantic Information

“…We extract latent topics from human labeled descriptions as semantic information and introduce an interpretive loss to guide the learning towards interpretable features, which is optimized jointly with the negative log-likelihood of training descriptions. et al [20,21] proposed one original method for joint human action modeling and grouping, which can provide comprehensive information for video caption modeling and explicitly benefit understanding what happens in the given video. As a video is more than a set of static images, in which there are not only the static objects but also the temporal relationships and actions, video analysis often requires more complex network architectures.…”

Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks

“…Accuracy SV FV iDT-Tra (BoW) [21] 69.8% 65.8% iDT-COM (BoW) [21] 76.9% 75.3% iDT-COM (FV) [21] 80.7% 79.5% iDT-MBH (BoW) [21] 77.2% 79.6% SFAM-D 71.2% 83.0% SFAM-S 70.1% 75.0% SFAM-RP 79.9% 81.8% SFAM-AMRP 82.2% 78.0% SFAM-LABRP 72.0% 83.7% Max-Score Fusion All 87.6% 88.8% Average-Score Fusion All 88.2% 89.1% Multiply-Score Fusion All 89.4% 91.2% Table 2: Comparison on the M 2 I Dataset for single task scenario (learning and testing in the same view).…”

“…Accuracy SV → FV FV → SV iDT-Tra [21] 43.3% 39.2% iDT-COM [21] 70.2% 67.7% iDT-HOG+MBH [21] 75.8% 71.8% iDT-HOG+HOF [21] 78 nel Transform Kernels, end-to-end, with ConvNets from data. Experiments on two benchmark datasets have demonstrated the effectiveness of the proposed method.…”

“…Recognition of human actions from RGB-D data has generated renewed interest in the computer vision com- * Corresponding author † To be appear in CVPR2017 Figure 1: Samples of variants of SFAM for action "Bounce Basketball" from M 2 I Dataset [21]. For top-left to bottomright, the images correspond to SFAM-D, SFAM-S, SFAM-RPf, SFAM-RPb, SFAM-AMRPf, SFAM-AMRPb, SFAM-LABRPf, SFAM-LABRPb.…”

Multi-Camera Action Dataset for Cross-Camera Action Recognition Benchmarking

“…In the existing literature, many datasets are often collected under single camera view [15,34] or multiple views with overlapped observation [29,30,50]. Hence, it is hard to systematically evaluate the robustness of action recognition algorithms on similar yet different backgrounds and captured environments.…”