Real-Time RGB-D Activity Prediction by Soft Regression

Hu, Jiefeng; Zheng, Wei‐Shi; Ma, Lihua; Wang, Gang; Lai, Jianhuang

doi:10.1007/978-3-319-46448-0_17

Cited by 80 publications

(66 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A preliminary version of this work was reported in [15]. In this work, we have further extended our soft regression-based early action prediction model in the following three aspects.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Early Action Prediction by Soft Regression

Zheng

et al. 2019

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

View full text Add to dashboard Cite

Abstract-We propose a novel approach for predicting on-going action with the assistance of a low-cost depth camera. Our approach introduces a soft regression-based early prediction framework. In this framework, we estimate soft labels for the subsequences at different progress levels, jointly learned with an action predictor. Our formulation of soft regression framework 1) overcomes a usual assumption in existing early action prediction systems that the progress level of on-going sequence is given in the testing stage; and 2) presents a theoretical framework to better resolve the ambiguity and uncertainty of subsequences at early performing stage. The proposed soft regression framework is further enhanced in order to take the relationships among subsequences and the discrepancy of soft labels over different classes into consideration, so that a Multiple Soft labels Recurrent Neural Network (MSRNN) is finally developed. For real-time performance, we also introduce a new RGB-D feature called "local accumulative frame feature (LAFF)", which can be computed efficiently by constructing an integral feature map. Our experiments on three RGB-D benchmark datasets and an unconstrained RGB action set demonstrate that the proposed regression-based early action prediction model outperforms existing models significantly and also show that the early action prediction on RGB-D sequence is more accurate than that on RGB channel.

show abstract

“…A preliminary version of this work was reported in [15]. In this work, we have further extended our soft regression-based early action prediction model in the following three aspects.…”

Section: Related Workmentioning

confidence: 99%

“…At step 2, we optimize it over α with W fixed. Please refer to our conference version [15] for more details.…”

Section: Soft Linear Regression (Slr) Based Early Action Predictionmentioning

confidence: 99%

Early Action Prediction by Soft Regression

Zheng

et al. 2019

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

View full text Add to dashboard Cite

show abstract

“…SYSU dataset: For the empirical evaluations, we compare our DACNN algorithm to other baselines including CNN+DPRL [28], ST-LSTM+Trust Gate [13], Dynamic Skeletons [35], LAFF(SKL) [45], SR-TSL [46], VA-LSTM [47], and GCA-LSTM [48], which includes the most recent deep learning applications (CNN, LSTM, etc.) on this dataset.…”

Section: Action Recognition Resultsmentioning

confidence: 99%

Deep-Aligned Convolutional Neural Network for Skeleton-Based Action Recognition and Segmentation

Hosseini

Montagne

Hammer

2019

2019 IEEE International Conference on Data Mining (ICDM)

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) are deep learning frameworks which are well-known for their notable performance in classification tasks. Hence, many skeleton-based action recognition and segmentation (SBARS) algorithms benefit from them in their designs. However, a shortcoming of such applications is the general lack of spatial relationships between the input features in such data types. Besides, non-uniform temporal scalings is a common issue in skeletonbased data streams which leads to having different input sizes even within one specific action category. In this work, we propose a novel deep-aligned convolutional neural network (DACNN) to tackle the above challenges for the particular problem of SBARS. Our network is designed by introducing a new type of filters in the context of CNNs which are trained based on their alignments to the local subsequences in the inputs. These filters result in efficient predictions as well as learning interpretable patterns in the data. We empirically evaluate our framework on real-world benchmarks showing that the proposed DACNN algorithm obtains a competitive performance compared to the state-of-the-art while benefiting from a less complicated yet more interpretable model. * Preprint of the publication [1] including extended experiments, as provided by the authors. The final publication is available at https://ieeexplore.ieee.org/ networks (RNN) [8,12,13]. RNN methods can learn the temporal dynamics of the sequential data; nevertheless, they have practical shortcomings in the training of their stacked structures [14,15].Compared to RNN architectures, CNN-based methods provide more effective solutions by extracting local features from their input and finding discriminative patterns in the data [16,10]. However, regardless of the promising feature extraction capability of CNN, its specific convolutional structure is designed originally for image-based input data and primarily relies on spatial dependencies between the neighboring points. In contrast, such a direct relationship does not generally exist in skeleton-based action datasets. Although some works tried to solve this problem by using 1-dimensional filters (only for the temporal dimension), it is still not an efficient solution to this specific shortcoming of CNN-based frameworks [17].A crucial step before analyzing any motion data is the temporal segmentation phase, in which we predict the action to which each time-frame belongs. Although there exist unsupervised algorithms for temporal segmentation of motion data [18,19], they usually oversegment actions into smaller sub-actions or segment also the blank parts of the stream.Motivations: An essential group of techniques for classification of the sequential data is time-series alignment methods [20]. It is shown that via comparing each data sequence to some predefined or learned sequences, we can discriminate or segment the data samples with high accuracy [21,22]. Also, in algorithms similar to [23], finding a small distinct subsequence in the input data can reveal i...

show abstract

“…Year Dynamic Skeletons [60] 75.5 2015 LAFF (SKL) [65] 54.2 2016 ST-LSTM (Tree) [23] 73.4 2018 ST-LSTM (Tree) + Trust Gate [23] 76.5 2018 DPRL [34] 76.9 2018 GR-GCN (Bone only) 75.2 Complete GR-GCN model 77.9…”

Section: Methods Accuracymentioning

confidence: 99%

Optimized Skeleton-based Action Recognition via Sparsified Graph Regression

Gao

Tang

et al. 2019

Proceedings of the 27th ACM International Conference on Multimedia

166

105

View full text Add to dashboard Cite

With the prevalence of accessible depth sensors, dynamic human body skeletons have attracted much attention as a robust modality for action recognition. Previous methods model skeletons based on RNN or CNN, which has limited expressive power for irregular skeleton joints. While graph convolutional networks (GCN) have been proposed to address irregular graphstructured data, the fundamental graph construction remains challenging. In this paper, we represent skeletons naturally on graphs, and propose a graph regression based GCN (GR-GCN) for skeleton-based action recognition, aiming to capture the spatio-temporal variation in the data. As the graph representation is crucial to graph convolution, we first propose graph regression to statistically learn the underlying graph from multiple observations. In particular, we provide spatio-temporal modeling of skeletons and pose an optimization problem on the graph structure over consecutive frames, which enforces the sparsity of the underlying graph for efficient representation. The optimized graph not only connects each joint to its neighboring joints in the same frame strongly or weakly, but also links with relevant joints in the previous and subsequent frames. We then feed the optimized graph into the GCN along with the coordinates of the skeleton sequence for feature learning, where we deploy high-order and fast Chebyshev approximation of spectral graph convolution. Further, we provide analysis of the variation characterization by the Chebyshev approximation. Experimental results validate the effectiveness of the proposed graph regression and show that the proposed GR-GCN achieves the state-of-the-art performance on the widely used NTU RGB+D, UT-Kinect and SYSU 3D datasets.

show abstract

Real-Time RGB-D Activity Prediction by Soft Regression

Cited by 80 publications

References 35 publications

Early Action Prediction by Soft Regression

Early Action Prediction by Soft Regression

Deep-Aligned Convolutional Neural Network for Skeleton-Based Action Recognition and Segmentation

Optimized Skeleton-based Action Recognition via Sparsified Graph Regression

Contact Info

Product

Resources

About