Multi-modal egocentric activity recognition using multi-kernel learning

Arabaci, Mehmet Ali; Özkan, Fatih; Sürer, Elif; Jančovič, Peter; Temizel, Alptekin

doi:10.1007/s11042-020-08789-7

Cited by 14 publications

(9 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lu et al [31] used LSTM to categorize activities utilizing four IMU sensors and egocentric video from the CMU Multimodal Activity (CMU-MMAC) database [32]. For activity recognition, [33] used visual and audio sensors. [8] described a methodology for detecting proprioceptive activities using egocentric data from IMUs.…”

Section: Related Work and Contributionsmentioning

confidence: 99%

Learning Human Activity From Visual Data Using Deep Learning

et al. 2021

View full text Add to dashboard Cite

Advances in wearable technologies have the ability to revolutionize and improve people's lives. The gains go beyond the personal sphere, encompassing business and, by extension, the global economy. The technologies are incorporated in electronic devices that collect data from consumers' bodies and their immediate environment. Human activities recognition, which involves the use of various body sensors and modalities either separately or simultaneously, is one of the most important areas of wearable technology development. In real-life scenarios, the number of sensors deployed is dictated by practical and financial considerations. In the research for this article, we reviewed our earlier efforts and have accordingly reduced the number of required sensors, limiting ourselves to first-person vision data for activities recognition. Nonetheless, our results beat state of the art by more than 4% of F1 score.

show abstract

Section: Related Work and Contributionsmentioning

confidence: 99%

Learning Human Activity From Visual Data Using Deep Learning

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Feature-level fusion is applied in [14] by combining audio-visual features with multi-kernel learning and multi-kernel boosting. After extracting specific features for each modality, the proposed framework performs adaptive fusion by selecting and weighting the features and kernels jointly.…”

Section: Multimodal Learningmentioning

confidence: 99%

Multimodal Deep Learning for Group Activity Recognition in Smart Office Environments

Florea

Mihailescu

2020

Future Internet

View full text Add to dashboard Cite

Deep learning (DL) models have emerged in recent years as the state-of-the-art technique across numerous machine learning application domains. In particular, image processing-related tasks have seen a significant improvement in terms of performance due to increased availability of large datasets and extensive growth of computing power. In this paper we investigate the problem of group activity recognition in office environments using a multimodal deep learning approach, by fusing audio and visual data from video. Group activity recognition is a complex classification task, given that it extends beyond identifying the activities of individuals, by focusing on the combinations of activities and the interactions between them. The proposed fusion network was trained based on the audio–visual stream from the AMI Corpus dataset. The procedure consists of two steps. First, we extract a joint audio–visual feature representation for activity recognition, and second, we account for the temporal dependencies in the video in order to complete the classification task. We provide a comprehensive set of experimental results showing that our proposed multimodal deep network architecture outperforms previous approaches, which have been designed for unimodal analysis, on the aforementioned AMI dataset.

show abstract

“…Lu and Velipasalar [25] used LSTM to classify actions using four IMUs sensors corresponding 36 components with egocentric video from CMU Multimodal Activity (CMU-MMAC) Database [41]. Visual and audio sensors was used by [5] for activity recognition. [1] has presented framework for recognizing proprioceptive activities using IMUs egocentric data.…”

Section: Related Workmentioning

confidence: 99%

Action Recognition Using Local Visual Descriptors and Inertial Data

Alhersh

Belhaouari

Stuckenschmidt

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Different body sensors and modalities can be used in human action recognition, either separately or simultaneously. Multi-modal data can be used in recognizing human action. In this work we are using inertial measurement units (IMUs) positioned at left and right hands with first person vision for human action recognition. A novel statistical feature extraction method was proposed based on curvature of the graph of a function and tracking left and right hand positions in space. Local visual descriptors have been used as features for egocentric vision. An intermediate fusion between IMUs and visual sensors has been performed. Despite of using only two IMUs sensors with egocentric vision, our classification result achieved is 99.61% for recognizing nine different actions. Feature extraction step could play a vital step in human action recognition with limited number of sensors, hence, our method might indeed be promising.

show abstract

Multi-modal egocentric activity recognition using multi-kernel learning

Cited by 14 publications

References 60 publications

Learning Human Activity From Visual Data Using Deep Learning

Learning Human Activity From Visual Data Using Deep Learning

Multimodal Deep Learning for Group Activity Recognition in Smart Office Environments

Action Recognition Using Local Visual Descriptors and Inertial Data

Contact Info

Product

Resources

About