Deep Contextual Attention for Human-Object Interaction Detection

Wang, Tiancai; Anwer, Rao Muhammad; Khan, Muhammad Haris; Khan, Fahad Shahbaz; Pang, Yanwei; Shao, Ling; Laaksonen, Jorma

doi:10.1109/iccv.2019.00579

Cited by 130 publications

(93 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The other edges are defined based on the consistencies among objects, actions and interactions. That is, if two [25] ResNet-50 48.7 BAR-CNN [21] Inception-ResNet 43.6 Wang et al [41] ResNet-50 47.3 PMFNet [39] ResNet-50 52.0 VSGNet [37] ResNet-152 51.8…”

Section: Semantic Embedding Networkmentioning

confidence: 99%

See 1 more Smart Citation

ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection

Liu

Yuan

Chen

2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

We consider the problem of Human-Object Interaction (HOI) Detection, which aims to locate and recognize HOI instances in the form of ⟨ℎ , , ⟩ in images. Most existing works treat HOIs as individual interaction categories, thus can not handle the problem of long-tail distribution and polysemy of action labels. We argue that multi-level consistencies among objects, actions and interactions are strong cues for generating semantic representations of rare or previously unseen HOIs. Leveraging the compositional and relational peculiarities of HOI labels, we propose ConsNet, a knowledge-aware framework that explicitly encodes the relations among objects, actions and interactions into an undirected graph called consistency graph, and exploits Graph Attention Networks (GATs) to propagate knowledge among HOI categories as well as their constituents. Our model takes visual features of candidate human-object pairs and word embeddings of HOI labels as inputs, maps them into visual-semantic joint embedding space and obtains detection results by measuring their similarities. We extensively evaluate our model on the challenging V-COCO and HICO-DET datasets, and results validate that our approach outperforms stateof-the-arts under both fully-supervised and zero-shot settings. CCS CONCEPTS • Computing methodologies → Activity recognition and understanding; Scene understanding.

show abstract

Section: Semantic Embedding Networkmentioning

confidence: 99%

“…Most existing works on HOI detection [9,11,14,25,36,39,41] treat HOIs as individual interaction categories and focus on mining visual representations of human-object pairs to improve classification performances. Despite previous successes, these conventional Figure 2: Polysemy of action labels.…”

Section: Introductionmentioning

confidence: 99%

ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection

Liu

Yuan

Chen

2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…Though impressive solutions have been devised using deep models for human action recognition, it is yet a challenging task to discriminate various fine-grained activities like playing a violin vs a guitar, using a phone vs talking to passengers while driving, etc. It can be regarded as a more challenging problem when multiple actions appear in a single image, such as walking and talking over the phone [8].…”

Section: Introductionmentioning

confidence: 99%

Attend and Guide (AG-Net): A Keypoints-Driven Attention-Based Deep Network for Image Recognition

Bera

Wharton

Liu

et al. 2021

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

This paper presents a novel keypoints-based attention mechanism for visual recognition in still images. Deep Convolutional Neural Networks (CNNs) for recognizing images with distinctive classes have shown great success, but their performance in discriminating fine-grained changes is not at the same level. We address this by proposing an end-to-end CNN model, which learns meaningful features linking fine-grained changes using our novel attention mechanism. It captures the spatial structures in images by identifying semantic regions (SRs) and their spatial distributions, and is proved to be the key to modelling subtle changes in images. We automatically identify these SRs by grouping the detected keypoints in a given image. The "usefulness" of these SRs for image recognition is measured using our innovative attentional mechanism focusing on parts of the image that are most relevant to a given task. This framework applies to traditional and fine-grained image recognition tasks and does not require manually annotated regions (e.g. boundingbox of body parts, objects, etc.) for learning and prediction. Moreover, the proposed keypoints-driven attention mechanism can be easily integrated into the existing CNN models. The framework is evaluated on six diverse benchmark datasets. The model outperforms the state-of-the-art approaches by a considerable margin using Distracted Driver V1 (Acc: 3.39%), Distracted Driver V2 (Acc: 6.58%), Stanford-40 Actions (mAP: 2.15%), People Playing Musical Instruments (mAP: 16.05%), Food-101 (Acc: 6.30%) and Caltech-256 (Acc: 2.59%) datasets.

show abstract

“…Most previous methods either rely on employing handcrafted visual features [13,24,47,77], such as color and shape, or using mid-level holistic image representations [11,33,44,75,76] constructed by encoding hand-crafted visual features. Recently, deep Convolutional Neural Networks (CNNs) have revolutionized computer vision by significantly advancing the state-of-the-art in many areas such as, image classification [8,20,31,38,43,64], object detection/segmentation [19,28,30,40,53,57,68,69,81] and action recognition [26,48,51,63]. Similarly, deep learning techniques have also made an impact on satellite image analysis, including aerial scene classification [2,32,54,73] and hyperspectral image analysis [14,15,66].…”

Section: Introductionmentioning

confidence: 99%

Compact Deep Color Features for Remote Sensing Scene Classification

Anwer

Khan

Laaksonen

2021

Neural Process Lett

Self Cite

View full text Add to dashboard Cite

Aerial scene classification is a challenging problem in understanding high-resolution remote sensing images. Most recent aerial scene classification approaches are based on Convolutional Neural Networks (CNNs). These CNN models are trained on a large amount of labeled data and the de facto practice is to use RGB patches as input to the networks. However, the importance of color within the deep learning framework is yet to be investigated for aerial scene classification. In this work, we investigate the fusion of several deep color models, trained using color representations, for aerial scene classification. We show that combining several deep color models significantly improves the recognition performance compared to using the RGB network alone. This improvement in classification performance is, however, achieved at the cost of a high-dimensional final image representation. We propose to use an information theoretic compression approach to counter this issue, leading to a compact deep color feature set without any significant loss in accuracy. Comprehensive experiments are performed on five remote sensing scene classification benchmarks: UC-Merced with 21 scene classes, WHU-RS19 with 19 scene types, RSSCN7 with 7 categories, AID with 30 aerial scene classes, and NWPU-RESISC45 with 45 categories. Our results clearly demonstrate that the fusion of deep color features always improves the overall classification performance compared to the standard RGB deep features. On the large-scale NWPU-RESISC45 dataset, our deep color features provide a significant absolute gain of 4.3% over the standard RGB deep features.

show abstract

Deep Contextual Attention for Human-Object Interaction Detection

Cited by 130 publications

References 28 publications

ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection

ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection

Attend and Guide (AG-Net): A Keypoints-Driven Attention-Based Deep Network for Image Recognition

Compact Deep Color Features for Remote Sensing Scene Classification

Contact Info

Product

Resources

About