Multimodal Multi-Stream Deep Learning for Egocentric Activity Recognition

Song, Shang; Chandrasekhar, Vijay; Mandal, Bappaditya; Li, Liyuan; Lim, Joo‐Hwee; Babu, G. S. D.; San, Phyo Phyo; Cheung, Ngai-Man

doi:10.1109/cvprw.2016.54

Cited by 67 publications

(74 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Human Activity Recognition is an active field of research in pervasive computing [1]- [3]. One popular task of this field is the recognition of so called activities of daily living [4].…”

Section: Introductionmentioning

confidence: 99%

“…We propose the usage of off-the-shelf smart-devices to recognize such activities, where we rely on inertial sensors and an ego-centric camera. Several studies already investigate activity recognition, be it low-level [2], [7] or high-level activities [3], [8]. Usually, the former comprises actions like walking where the latter refers to context-enriched actions such as preparing food.…”

Section: Introductionmentioning

confidence: 99%

“…Approaches for fusing inertial and vision sensor have been made by other researchers [10]. However, most of the work focuses on the fusion of sensor streams that belong to the same on-body position [3], [11]. In this paper, we present a multi-modal ego-centric activity recognition approach that relies on smart-watches and smart-glasses to recognize highlevel activities like activities of daily living.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Vision and Acceleration Modalities: Partners for Recognizing Complex Activities

Diete

Sztyler

Stuckenschmidt

2019

2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops)

View full text Add to dashboard Cite

Wearable devices have been used widely for human activity recognition in the field of pervasive computing. One big area of in this research is the recognition of activities of daily living where especially inertial and interaction sensors like RFID tags and scanners have been used. An issue that may arise when using interaction sensors is a lack of certainty. A positive signal from an interaction sensor is not necessarily caused by a performed activity e.g. when an object is only touched but no interaction occurred afterwards. In our work, we aim to overcome this limitation and present a multi-modal egocentricbased activity recognition approach which is able to recognize the critical activities by looking at movement and object information at the same time. We present our results of combining inertial and video features to recognize human activities on different types of scenarios where we achieve a F1-measure up to 79.6%.

show abstract

“…Human Activity Recognition is an active field of research in pervasive computing [1]- [3]. One popular task of this field is the recognition of so called activities of daily living [4].…”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Vision and Acceleration Modalities: Partners for Recognizing Complex Activities

Diete

Sztyler

Stuckenschmidt

2019

2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops)

View full text Add to dashboard Cite

show abstract

“…1). DNN has been applied to many image/video applications [8], [9], [10], [11], [12]. Whilet these methods achieve state-of-the-art on various datasets [13], direct application of them requires a large dataset, that is laborious and expensive to construct.…”

Section: Introductionmentioning

confidence: 99%

Image-based vehicle analysis using deep neural network: A systematic study

Zhou

Nejati

et al. 2016

2016 IEEE International Conference on Digital Signal Processing (DSP)

Self Cite

View full text Add to dashboard Cite

Abstract-We address the vehicle detection and classification problems using Deep Neural Networks (DNNs) approaches. Here we answer to questions that are specific to our application including how to utilize DNN for vehicle detection, what features are useful for vehicle classification, and how to extend a model trained on a limited size dataset, to the cases of extreme lighting condition. Answering these questions we propose our approach that outperforms state-of-the-art methods, and achieves promising results on image with extreme lighting conditions.

show abstract

“…In first-person activity recognition, there are DCNN-based methods in which optical flow [5,6] and pooled motion features [33] are used as image features. Moreover, LSTM (long short-term memory) model, which is also a kind of deep models for learning data with recursive expressions, has been introduced together with DCNN models aiming at additionally learning temporal correlations of activities [7,8].…”

Section: Deep Convolutional Neural Networkmentioning

confidence: 99%

First-person reading activity recognition by deep learning with synthetically generated images

Segawa¹,

Kawamoto

Okamoto

2018

J Image Video Proc.

View full text Add to dashboard Cite

We propose a vision-based method for recognizing first-person reading activity with deep learning. For the success of deep learning, it is well known that a large amount of training data plays a vital role. Unlike image classification, there are less publicly available datasets for reading activity recognition, and the collection of book images might cause copyright trouble. In this paper, we develop a synthetic approach for generating positive training images. Our approach synthesizes computer-generated images and real backround images. In experiments, we show that this synthesis is effective in combination with pre-trained deep convolutional neural networks and also our trained neural network outperforms other baselines.

show abstract

Multimodal Multi-Stream Deep Learning for Egocentric Activity Recognition

Cited by 67 publications

References 19 publications

Vision and Acceleration Modalities: Partners for Recognizing Complex Activities

Vision and Acceleration Modalities: Partners for Recognizing Complex Activities

Image-based vehicle analysis using deep neural network: A systematic study

First-person reading activity recognition by deep learning with synthetically generated images

Contact Info

Product

Resources

About