Extensible Hierarchical Method of Detecting Interactive Actions for Video Understanding

Abstract

For video understanding, namely analyzing who did what in a video, actions along with objects are primary elements. Most studies on actions have handled recognition problems for a well-trimmed video and focused on enhancing their classification performance. However, action detection, including localization as well as recognition, is required because, in general, actions intersect in time and space. In addition, most studies have not considered extensibility for a newly added action that has been previously trained. Therefore, proposed in this paper is an extensible hierarchical method for detecting generic actions, which combine object movements and spatial relations between two objects, and inherited actions, which are determined by the related objects through an ontology and rule based methodology. The hierarchical design of the method enables it to detect any interactive actions based on the spatial relations between two objects. The method using object information achieves an F-measure of 90.27%. Moreover, this paper describes the extensibility of the method for a new action contained in a video from a video domain that is different from the dataset used.

Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning

“…As wearable devices are widely used nowadays, Human Activity Recognition (HAR) has become an emerging research area in mobile and wearable computing. Recognizing human activity leads to a deep understanding of individuals’ activity patterns or long-term habits, which contributes to developing a wide range of user-centric applications such as human-computer interaction [1,2], surveillance [3,4], video streaming [5,6], AR/VR [7], and healthcare systems [8,9]. Although activity recognition has been investigated to a great extent in computer vision area [10,11,12], the application is limited to a certain scenario that equips pre-installed cameras with sufficient resolution and guaranteed angle of view.…”

RGB-D Data-based Action Recognition: A Review

“…To resolve this issue, a two-stream CNNs model was proposed by Simnonyan and Zisserman [27] to explicitly take into account both spatial and temporal information in a single end-to-end learning framework. [28] ActivityNet [29] ActionNet-VE [30] UCF101 [31] THUMOS'14 [32] Something-Something v2 [33] VIVA Hand Gestures [34] UTD-MHAD [35] EgoGesture [36] AVA 2.1 [37] Something-Something v1 [38] Charades [39] NTU RGB+D 120 [40] miniSports [41] Sports-1M [26] IRD [42,43] HMDB-51 [44] ICVL-4 [42,43] NTU RGB+D 60 [45] Jester [46] 2014 [47] PDF 2016 [48] LSTM 75 2017 [49] Ontology/Rule 90 2017 [50] i3D CNN 98 2018 [51] CNN 37 2018 [33] 2D CNN 66 2018 [52] 3D CNN 86 93 2018 [43] i3D CNN 92 94 2019 [53] iD CNN+Attention 27 2019 [38] 2D…”

“…Stacked Fisher vectors were used by Peng et al [79] that further improved iDT. However, Wang et al [47] have used a variational Bayes method, while Moon et al [49], through an ontology and rule-based methodology, have produced benchmark action recognition techniques on RGB-based datasets.…”

Zero-Shot Human Activity Recognition Using Non-Visual Sensors

“…Furthermore, activity recognition has been widely reported in many fields using sensor modalities, including ambient sensors [35], wearable sensors [36], smart phones [34], and smart watches [37]. Those sensors contribute to developing a wide range of application domains such as sport [38], human-computer interaction [39], surveillance [40], video streaming [41], healthcare system [42], and computer vision area [43]. Due to the properties of noninvasive sensors, some studies discussed how to monitor human activities using this type of sensors (i.e., non-visual sensors) because they are both easy to install and privacy preserving [44,45].…”