Exemplar-Based Recognition of Human–Object Interactions

IEEE Trans. Pattern Anal. Mach. Intell.

et al. 2019

Self Cite

Abstract-We propose a novel approach for predicting on-going action with the assistance of a low-cost depth camera. Our approach introduces a soft regression-based early prediction framework. In this framework, we estimate soft labels for the subsequences at different progress levels, jointly learned with an action predictor. Our formulation of soft regression framework 1) overcomes a usual assumption in existing early action prediction systems that the progress level of on-going sequence is given in the testing stage; and 2) presents a theoretical framework to better resolve the ambiguity and uncertainty of subsequences at early performing stage. The proposed soft regression framework is further enhanced in order to take the relationships among subsequences and the discrepancy of soft labels over different classes into consideration, so that a Multiple Soft labels Recurrent Neural Network (MSRNN) is finally developed. For real-time performance, we also introduce a new RGB-D feature called "local accumulative frame feature (LAFF)", which can be computed efficiently by constructing an integral feature map. Our experiments on three RGB-D benchmark datasets and an unconstrained RGB action set demonstrate that the proposed regression-based early action prediction model outperforms existing models significantly and also show that the early action prediction on RGB-D sequence is more accurate than that on RGB channel.

Section: A Soft Rnn (Srnn) Regression Based Early Action Predictionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Early Action Prediction by Soft Regression

IEEE Trans. Pattern Anal. Mach. Intell.

et al. 2019

Self Cite

“…Compared to the conventional use of RGB videos, the information from depth channel is insensitive to illumination variations, invariant to color and texture changes, and more importantly reliable for body silhouette and skeleton (human posture) extraction [29]. Bearing on these merits, it is believed that the introduced depth information can greatly light up the research of human activity analysis [10], [23], [34], [48].…”

Section: Introductionmentioning

confidence: 99%

“…Moreover, the depth information is usually weak in capturing the important appearance information, such as object color and texture. These greatly limit the application of depth cameras on recognizing complex human activities with object and interactions, such as human-object interactions [10] and fine grained activities [15], where the color appearance is also important.…”

Section: Introductionmentioning

confidence: 99%

Jointly learning heterogeneous features for RGB-D activity recognition

2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Lai

et al. 2015

Self Cite

324

209

In this paper, we focus on heterogeneous features learning for RGB-D activity recognition. We find that features from different channels (RGB, depth) could share some similar hidden structures, and then propose a joint learning model to simultaneously explore the shared and feature-specific components as an instance of heterogeneous multi-task learning. The proposed model formed in a unified framework is capable of: 1) jointly mining a set of subspaces with the same dimensionality to exploit latent shared features across different feature channels, 2) meanwhile, quantifying the shared and feature-specific components of features in the subspaces, and 3) transferring feature-specific intermediate transforms (i-transforms) for learning fusion of heterogeneous features across datasets. To efficiently train the joint model, a three-step iterative optimization algorithm is proposed, followed by a simple inference model. Extensive experimental results on four activity datasets have demonstrated the efficacy of the proposed method. A new RGB-D activity dataset focusing on human-object interaction is further contributed, which presents more challenges for RGB-D activity benchmarking.

“…The research challenges and focus on collective activity recognition should differ significantly from the widely studied action recognition [21], [28], [31], [32], [35], [37], [41]- [43], where the actions performed by individuals are the main focus. It should also be distinguished from the crowd activity recognition [19], [27], [29], [30] in a way that collective activity is not to model a crowd scenario but rather to infer collective person-person interactions between several people.…”

mentioning

confidence: 99%

Learning Person–Person Interaction in Collective Activity Recognition

Chang

IEEE Trans. on Image Process.

Zhang

2015

Collective activity is a collection of atomic activities (individual person's activity) and can hardly be distinguished by an atomic activity in isolation. The interactions among people are important cues for recognizing collective activity. In this paper, we concentrate on modeling the person-person interactions for collective activity recognition. Rather than relying on hand-craft description of the person-person interaction, we propose a novel learning-based approach that is capable of computing the class-specific person-person interaction patterns. In particular, we model each class of collective activity by an interaction matrix, which is designed to measure the connection between any pair of atomic activities in a collective activity instance. We then formulate an interaction response (IR) model by assembling all these measurements and make the IR class specific and distinct from each other. A multitask IR is further proposed to jointly learn different person-person interaction patterns simultaneously in order to learn the relation between different person-person interactions and keep more distinct activity-specific factor for each interaction at the same time. Our model is able to exploit discriminative low-rank representation of person-person interaction. Experimental results on two challenging data sets demonstrate our proposed model is comparable with the state-of-the-art models and show that learning person-person interactions plays a critical role in collective activity recognition.