NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis

Abstract

Recent approaches in depth-based human activity analysis achieved outstanding performance and proved the effectiveness of 3D representation for classification of action classes. Currently available depth-based and RGB+Dbased action recognition benchmarks have a number of limitations, including the lack of training samples, distinct class labels, camera views and variety of subjects. In this paper we introduce a large-scale dataset for RGB+D human action recognition with more than 56 thousand video samples and 4 million frames, collected from 40 distinct subjects. Our dataset contains 60 different action classes including daily, mutual, and health-related actions. In addition, we propose a new recurrent neural network structure to model the long-term temporal correlation of the features for each body part, and utilize them for better action classification. Experimental results show the advantages of applying deep learning methods over state-of-the-art handcrafted features on the suggested cross-subject and crossview evaluation criteria for our dataset. The introduction of this large scale dataset will enable the community to apply, develop and adapt various data-hungry learning techniques for the task of depth-based and RGB+D-based human activity analysis.

Evaluation of a skeleton-based method for human activity recognition on a large-scale RGB-D dataset

“…A multi-class SVM is the considered classification algorithm. The proposed method has been evaluated on the largescale NTU RGB+D dataset [17], reaching results comparable to other approaches exploiting skeleton-based features.…”

“…The algorithm performance is evaluated on the NTU RGB+D dataset [17], which is at the authors' best knowledge, the largest dataset for 3D action recognition currently available. It has been captured to overcome the limitations of the existing datasets, providing more data that are required to develop algorithms closer to real conditions.…”

“…Shahroudy et al [17] defined two evaluation methods, to enable a fair comparison of different activity recognition algorithms:…”

Extension of the Action Verb Corpus for Supervised Learning

“…The Action Verb Corpus with its extension will be made available to the scientific community alongside this publication 2 . At the point of writing the dataset consists of 210 annotated TAKE/PUT and 100 PUSH actions.…”

“…The extension presented in this paper is focused on supervised learning for action recognition from human motion data. Existing datasets for action recognition that provide skeleton tracking often use the Microsoft Kinect camera such as the NTU RGB+D dataset [2] or the Montalbano dataset [3]. The Kinect tracks the whole-body skeleton but lacks individual finger tracking.…”