Recognizing Fine-Grained and Composite Activities Using Hand-Centric Features and Script Data

Abstract

Activity recognition has shown impressive progress in recent years. However, the challenges of detecting fine-grained activities and understanding how they are combined into composite activities have been largely overlooked. In this work we approach both tasks and present a dataset which provides detailed annotations to address them. The first challenge is to detect fine-grained activities, which are defined by low interclass variability and are typically characterized by finegrained body motions. We explore how human pose and hands can help to approach this challenge by comparing two pose-based and two hand-centric features with state-of-the-art holistic features. To attack the second challenge, recognizing composite activities, we leverage the fact that these activities are compositional and that the essential components of the activities can be obtained from textual descriptions or scripts.We show the benefits of our hand-centric approach for fine-grained activity classification and detection. For composite activity recognition we find that decomposition into attributes allows sharing information across composites and is essential to attack this hard task. Using script data we can recognize novel composites without having

Learning Activity Progression in LSTMs for Activity Detection and Early Detection

“…In [10,22], human actions are modeled as space-time structures, using deformable part models [2]. In [15,18] discriminative handcentric features are explored for fine grained activity detection in cooking, i.e., relatively short sub-activities such as chop and fill. In [3], the detector is trained on CNN features extracted from the action tubes in space-time; however, evaluation is on relatively short video clips (i.e., several hundred frames) of relatively short actions.…”

Action Recognition With Spatial-Temporal Discriminative Filter Banks

“…Understanding human activities as a fine-grained recognition problem has been explored for some domain specific tasks [20,24]. For example, some works have been proposed for hand-gesture recognition [10,17,22], daily life activity recognition [21] and sports understanding [7,27,3,1]. All these works build ad hoc solutions specific to the action domain they are addressing.…”

Coupled Generative Adversarial Network for Continuous Fine-Grained Action Segmentation

“…Therefore, using only frame wise metrics is insufficient to fully describe performance. Considering this, we also use the segmentation metrics: mean average precision with midpoint hit criterion (mAP@mid) [45,39], Segmental F1 score (F1@k) [23] and segmental edit score (edit) [25].…”