Modeling Sub-Event Dynamics in First-Person Action Recognition

Zaki, Hasan Firdaus Mohd; Mian, Ajmal

doi:10.1109/cvpr.2017.176

Cited by 27 publications

(10 citation statements)

References 50 publications

(118 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another line of research consists of developing effective techniques for extracting temporal information present in the video. In [48], [73] features are extracted from a series of frames to perform temporal pooling with different operations, including max pooling, sum pooling, or histogram of gradients. Then, a temporal pyramid structure allows the encoding of both long term and short term characteristics.…”

Section: First Person Action Recognitionmentioning

confidence: 99%

Learning to Recognize Actions on Objects in Egocentric Video With Attention Dictionaries

Sudhakaran

Escalera

Lanz

2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

We present EgoACO, a deep neural architecture for video action recognition that learns to pool action-context-object descriptors from frame level features by leveraging the verb-noun structure of action labels in egocentric video datasets. The core component of EgoACO is class activation pooling (CAP), a differentiable pooling operation that combines ideas from bilinear pooling for fine-grained recognition and from feature learning for discriminative localization. CAP uses self-attention with a dictionary of learnable weights to pool from the most relevant feature regions. Through CAP, EgoACO learns to decode object and scene context descriptors from video frame features. For temporal modeling in EgoACO, we design a recurrent version of class activation pooling termed Long Short-Term Attention (LSTA). LSTA extends convolutional gated LSTM with built-in spatial attention and a redesigned output gate. Action, object and context descriptors are fused by a multi-head prediction that accounts for the inter-dependencies between noun-verb-action structured labels in egocentric video datasets. EgoACO features built-in visual explanations, helping learning and interpretation. Results on the two largest egocentric action recognition datasets currently available, EPIC-KITCHENS and EGTEA, show that by explicitly decoding action-context-object descriptors, EgoACO achieves state-of-the-art recognition performance.

show abstract

Section: First Person Action Recognitionmentioning

confidence: 99%

Learning to Recognize Actions on Objects in Egocentric Video With Attention Dictionaries

Sudhakaran

Escalera

Lanz

2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Fully convolutional approaches viewed action recognition as a learning-based problem with CNNs being used as appearance [24], [57] and motion [58] feature extractors. More data hungry methods used multi-stream deep networks that utilized optical flow alongside RGB images as input modalities [21], [38], [59], [60], [61] to be able to focus on motion.…”

Section: Video Activity Recognitionmentioning

confidence: 99%

Multi-Dataset, Multitask Learning of Egocentric Vision Tasks

Kapidis

Poppe

Veltkamp

2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

For egocentric vision tasks such as action recognition, there is a relative scarcity of labeled data. This increases the risk of overfitting during training. In this paper, we address this issue by introducing a multitask learning scheme that employs related tasks as well as related datasets in the training process. Related tasks are indicative of the performed action, such as the presence of objects and the position of the hands. By including related tasks as additional outputs to be optimized, action recognition performance typically increases because the network focuses on relevant aspects in the video. Still, the training data is limited to a single dataset because the set of action labels usually differs across datasets. To mitigate this issue, we extend the multitask paradigm to include datasets with different label sets. During training, we effectively mix batches with samples from multiple datasets. Our experiments on egocentric action recognition in the EPIC-Kitchens, EGTEA Gaze+, ADL and Charades-EGO datasets demonstrate the improvements of our approach over single-dataset baselines. On EGTEA we surpass the current state-of-the-art by 2.47%. We further illustrate the cross-dataset task correlations that emerge automatically with our novel training scheme.

show abstract

“…These features include the location of hands [30]- [32], the interaction with active/passive objects [33]- [38], the head motion [39], [40], the gaze [41]- [45], or a combination of them [46]- [48]. Other methods have explored egocentric contexts like social interactions [49] and the temporal structure of the activities [50]- [52]. Additionally, some approaches have adapted deep third-person action recognition methods [53]- [55] and developed new ones based on reinforcement learning [7].…”

Section: B Activity Recognition From Egocentric Videosmentioning

confidence: 99%

Activities of Daily Living Monitoring via a Wearable Camera: Toward Real-World Applications

2020

View full text Add to dashboard Cite

Activity recognition from wearable photo-cameras is crucial for lifestyle characterization and health monitoring. However, to enable its wide-spreading use in real-world applications, a high level of generalization needs to be ensured on unseen users. Currently, state-of-the-art methods have been tested only on relatively small datasets consisting of data collected by a few users that are partially seen during training. In this paper, we built a new egocentric dataset acquired by 15 people through a wearable photo-camera and used it to test the generalization capabilities of several state-of-the-art methods for egocentric activity recognition on unseen users and daily image sequences. In addition, we propose several variants to state-of-the-art deep learning architectures, and we show that it is possible to achieve 79.87% accuracy on users unseen during training. Furthermore, to show that the proposed dataset and approach can be useful in real-world applications, where data can be acquired by different wearable cameras and labeled data are scarcely available, we employed a domain adaptation strategy on two egocentric activity recognition benchmark datasets. These experiments show that the model learned with our dataset, can easily be transferred to other domains with a very small amount of labeled data. Taken together, those results show that activity recognition from wearable photo-cameras is mature enough to be tested in real-world applications. INDEX TERMS Daily activity recognition, visual lifelogs, domain adaptation, wearable cameras.

show abstract

Modeling Sub-Event Dynamics in First-Person Action Recognition

Cited by 27 publications

References 50 publications

Learning to Recognize Actions on Objects in Egocentric Video With Attention Dictionaries

Learning to Recognize Actions on Objects in Egocentric Video With Attention Dictionaries

Multi-Dataset, Multitask Learning of Egocentric Vision Tasks

Activities of Daily Living Monitoring via a Wearable Camera: Toward Real-World Applications

Contact Info

Product

Resources

About