Action Recognition Using Local Visual Descriptors and Inertial Data

Abstract

Different body sensors and modalities can be used in human action recognition, either separately or simultaneously. Multi-modal data can be used in recognizing human action. In this work we are using inertial measurement units (IMUs) positioned at left and right hands with first person vision for human action recognition. A novel statistical feature extraction method was proposed based on curvature of the graph of a function and tracking left and right hand positions in space. Local visual descriptors have been used as features for egocentric vision. An intermediate fusion between IMUs and visual sensors has been performed. Despite of using only two IMUs sensors with egocentric vision, our classification result achieved is 99.61% for recognizing nine different actions. Feature extraction step could play a vital step in human action recognition with limited number of sensors, hence, our method might indeed be promising.

Learning Human Activity From Visual Data Using Deep Learning

“…The research presented in the current article extends our work in [9], in which we used two sensors data (visual and inertial). Local visual descriptors were used as features for ego-centric vision data, while statistical features extracted from tracking left and right hand positions in space were used.…”

“…Local visual descriptors were used as features for ego-centric vision data, while statistical features extracted from tracking left and right hand positions in space were used. To overcome the drawbacks we analyzed in our previous work [9], i.e. heavy calculations for feature extraction and the time-consuming nature of the overall effort associated with multi-sensor data, we modified the experimental setup.…”

“…In order to improve human activity recognition, previous research recommended merging several sensor modalities [6]- [10]. The current article is based on new research that builds on our previous work [9], [10]. Since the original publication, we have come to the conclusion that the practical constraints of real-life scenarios dictate the use of fewer sensors with less complicated feature extraction method and faster execution time.…”