Vision and Acceleration Modalities: Partners for Recognizing Complex Activities

Abstract

Wearable devices have been used widely for human activity recognition in the field of pervasive computing. One big area of in this research is the recognition of activities of daily living where especially inertial and interaction sensors like RFID tags and scanners have been used. An issue that may arise when using interaction sensors is a lack of certainty. A positive signal from an interaction sensor is not necessarily caused by a performed activity e.g. when an object is only touched but no interaction occurred afterwards. In our work, we aim to overcome this limitation and present a multi-modal egocentricbased activity recognition approach which is able to recognize the critical activities by looking at movement and object information at the same time. We present our results of combining inertial and video features to recognize human activities on different types of scenarios where we achieve a F1-measure up to 79.6%.

Vision and Inertial Sensing Fusion for Human Action Recognition: A Review

“…In [119]- [121], an action recognition approach was introduced improving the motion-based action recognition with egocentric vision. In [120], [121], inertial data were collected from a smart-watch and video data were collected from a pair of smart-glasses to recognize actions. The inertial data were used to characterize the forearm movement pattern whereas the egocentric video data was used to characterize objects.…”

“…In [53], [120], [121], both feature-level fusion and decisionlevel fusion were examined with the decision-level fusion achieving higher accuracy. For the feature-level fusion, after combining the features from the two modalities, a softmax classifier was used to make the final decision.…”

“…For the decision-level fusion, two softmax classifiers were used to make two decision separately and then a fused decision was done via the Borda count method. In [120], [121], a Random Forest classifier was used for the acceleration data and Logistic Regression for the vision data. In [124], a tendimensional feature vector was first formed using statistical features from the acceleration signals.…”

EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding

“…IMU measures motion from an accelerometer and gyroscope and is widely available on popular wearable devices. Prior work leverages IMU as an extra modality for human action recognition [13,68,69], (e.g., jumping, walking, standing) or as geometric cues for visualinertial odometry [7,20,71].…”

Fusing Object Information and Inertial Data for Activity Recognition

“…This work is an extension of a previous publication [19]. In this extension, we included experiments with more data, a deeper analysis and a comparison of our work against another system.…”