Learning to Recognize Actions on Objects in Egocentric Video With Attention Dictionaries

Sudhakaran, Swathikiran; Escalera, Sérgio; Lanz, Oswald

doi:10.1109/tpami.2021.3058649

Cited by 14 publications

(7 citation statements)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In EK55 each action is described by the composition of a noun and a verb. To take into account such label structure both in prediction and for learning, we follow the "multihead prediction" three branches design of [57].…”

Section: A Details On Multi-label Predictionmentioning

confidence: 99%

Higher Order Recurrent Space-Time Transformer for Video Action Prediction

Tai,

Fiameni,

Lee

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Endowing visual agents with predictive capability is a key step towards video intelligence at scale. The predominant modeling paradigm for this is sequence learning, mostly implemented through LSTMs. Feed-forward Transformer architectures have replaced recurrent model designs in ML applications of language processing and also partly in computer vision. In this paper we investigate on the competitiveness of Transformer-style architectures for video predictive tasks. To do so we propose HORST, a novel higher order recurrent layer design whose core element is a spatial-temporal decomposition of self-attention for video. HORST achieves state of the art competitive performance on Something-Something-V2 early action recognition and EPIC-Kitchens-55 action anticipation, without exploiting a task specific design. We believe this is promising evidence of causal predictive capability that we attribute to our recurrent higher order design of self-attention.

show abstract

Section: A Details On Multi-label Predictionmentioning

confidence: 99%

Higher Order Recurrent Space-Time Transformer for Video Action Prediction

Tai,

Fiameni,

Lee

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…By means of the seminal works [24,25], deep neural networks are used for extracting features with high representational capacity. Sudhakaran et al [26] presented EgoACO for video action recognition that learns to pool action-context-object descriptors from frame level features by leveraging the verb-noun structure of action labels. By the aid of the multiscale feature maps output by a fully convolutional network, features of each individual within consecutive frames are fused together by a recurrent network [17].…”

Section: Descriptor Learning Without Interactionmentioning

confidence: 99%

Multiperson Interactive Activity Recognition Based on Interaction Relation Model

Chi

et al. 2021

Journal of Mathematics

View full text Add to dashboard Cite

Multiperson activity recognition is a pivotal branch as well as a challenging topic of human action recognition research. This paper adopts a hybrid learning model to the spatio-temporal relationship and occlusion relationship among multiple people. Initially, this paper builds up an active multiperson interaction relationship estimation framework model to capture interpersonal spatio-temporal relation. This model incorporates the interaction relationship estimation framework with the multiperson relationship network. On this ground, it automatically learns from the human-computer interaction dataset in an end-to-end manner and performs reasoning with standard matrix operations. Secondly, this paper proposed an adaptive occlusion state behavior recognition method derived from the semantic knowledge model to ravel out the concern of occlusion and self-occlusion in human action recognition. Then, Petri Nets are used to recognize multiperson interactive actions. This model has been through extensive experiments on the TV interaction dataset, Vlog dataset, AVA dataset, and MLB-YouTube dataset, experimental results have proved that the recognition performance of this model is superior than the other available models. This paper prospects and summarizes the estimation framework of the interaction relationship and occlusion semantic-knowledge relationship. Experimental results suggest that the proposed method in the paper could capture the discriminative relation information for multiperson interactive activity recognition, which further validates the efficiency of the hybrid learning model.

show abstract

“…Also, a framework using Long Short-Term Memory (LSTM) is presented where no scene labelling is necessary but as we can see in figure 2.9, objects are filtered depending on the scene. Finally, before introducing the object detection methods, it is worth mentioning EgoACO [29] where authors present a deep learning architecture to recognize actions on objects in egocentric videos. From class activation pooling using attention dictionaries the most relevant feature regions are used to decode object and scene descriptors.…”

Section: Object and Hand Recognition Approachesmentioning

confidence: 99%

Interaction Estimation in Egocentric Videos via Simultaneous Hand-Object Recognition

Benavent-Lledo

2021

Advances in Intelligent Systems and Computing

View full text Add to dashboard Cite

In this project, we propose an action estimation pipeline based on the simultaneous recognition of the hands and the objects in the scene from an egocentric perspective. To this end, research on the state of the art in this field has been conducted. From the latest approaches, we have come to the conclusion that the hands are a key element from this point of view. An action consists of the interactions of the hands with the different objects in the scene therefore, the 2D positions of hands and objects are used to compute the object which is more likely to be used.The architecture used for achieving this goal is YOLO, as its prediction speed allows us to predict the actions fluently with good accuracy on the detected objects and hands. After reviewing the available datasets and generators for hand and object detection, different experiments have been conducted. The best results determined by PascalVOC metric have been used in the proposed pipeline. Contents List of Figures xiii List of Tables xv List of Acronyms xviiList of Acronyms ADL Activities of Daily Living. AP Average Precision. API Application Program Interface. AR Average Recall. ASIC Application-Specific Integrated Circuit. CNN Convolutional Neural Network. CNTK Microsoft Cognitive Toolkit. CPU Central Processing Unit. CUDA Compute Unified Device Architecture. DPM Deformable Parts Model. DTIC Departamento de Tecnología Informática y Computación. EGTEA Extended Georgia Tech Egocentric Activity. FAIR Facebook Artificial Intelligence Group. GPU Graphics Processing Unit. IOU Intersection Over Union. LSTM Long Short-Term Memory. mAP mean Average Precision. mAR mean Average Recall. MLP Multilayer perceptron. MoDeAsS Monitoring and Detection of human behaviors for personalized assistance and early disease detection, A2HUMPA in Spanish. NAS Network Attached Storage. OOP Object Oriented Programming. RAID Redundant Array of Independent Disks. RPN Region Proposal Network. SSH Secure Shell. TPU Tensor Processing Unit. VMs Virtual Machines. YOLO You Only Look Once.

show abstract

Learning to Recognize Actions on Objects in Egocentric Video With Attention Dictionaries

Cited by 14 publications

References 70 publications

Higher Order Recurrent Space-Time Transformer for Video Action Prediction

Higher Order Recurrent Space-Time Transformer for Video Action Prediction

Multiperson Interactive Activity Recognition Based on Interaction Relation Model

Interaction Estimation in Egocentric Videos via Simultaneous Hand-Object Recognition

Contact Info

Product

Resources

About