Cross-Modality Attention with Semantic Graph Embedding for Multi-Label Classification

You, Renchun; Guo, Zhiyao; Cui, Lei; Long, Xiang; Bao, Yingze; Wen, Shifeng

doi:10.1609/aaai.v34i07.6964

Cited by 144 publications

(86 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Chen et al [20] used a GCN to build a graph of labels to represent a set of mutually dependent object classifiers. You et al [21] followed in this manner, but used a cross-modality attention mechanism instead. Additionally, there are some other works regarding multi-label classification.…”

Section: Label Semantic Relationshipsmentioning

confidence: 99%

A Serial Attention Frame for Multi-Label Waste Bottle Classification

Xiao

Xu²,

Tian

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

The multi-label recognition of damaged waste bottles has important significance in environmental protection. However, most of the previous methods are known for their poor performance, especially in regards to damaged waste bottle classification. In this paper, we propose the use of a serial attention frame (SAF) to overcome the mentioned drawback. The proposed network architecture includes the following three parts: a residual learning block (RB), a mixed attention block (MAB), and a self-attention block (SAB). The RB uses ResNet to pretrain the SAF to extract more detailed information. To address the effect of the complex background of waste bottle recognition, a serial attention mechanism containing MAB and SAB is presented. MAB is used to extract more salient category information via the simultaneous use of spatial attention and channel attention. SAB exploits the obtained features and its parameters to enable the diverse features to improve the classification results of waste bottles. The experimental results demonstrate that our proposed model exhibited good recognition performance in the collected waste bottle datasets, with eight labels of three classifications, i.e., the color, whether the bottle was damage, and whether the wrapper had been removed, as well as public image classification datasets.

show abstract

Section: Label Semantic Relationshipsmentioning

confidence: 99%

A Serial Attention Frame for Multi-Label Waste Bottle Classification

Xiao

Xu²,

Tian

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…Furthermore, Chen et al [28] proposed a semantic decoupling method which extracts label specific features and their dependence at the same time. Lastly, You et al [29] proposed an adjacency-based method to extract dependence associated to a cross-modality attention mechanism. Despite their wide success, these methods are specifically designed for computer vision tasks, thus we do not consider them further in this work.…”

Section: Multi-label Classificationmentioning

confidence: 99%

Deep tree-ensembles for multi-output prediction

Nakano

Pliakos

Vens

2022

Pattern Recognition

View full text Add to dashboard Cite

“…References [39][40][41] used the common-sense or structured prior knowledge to improve the performance of deep models. Other works, like References [42][43][44], tried to use graph embedding to learn some prior knowledge or relationships between label.…”

Section: Remote Sensing With Gnnmentioning

confidence: 99%

Knowledge and Spatial Pyramid Distance-Based Gated Graph Attention Network for Remote Sensing Semantic Segmentation

Cui

Meng

et al. 2021

Remote Sensing

View full text Add to dashboard Cite

The pixel-based semantic segmentation methods take pixels as recognitions units, and are restricted by the limited range of receptive fields, so they cannot carry richer and higher-level semantics. These reduce the accuracy of remote sensing (RS) semantic segmentation to a certain extent. Comparing with the pixel-based methods, the graph neural networks (GNNs) usually use objects as input nodes, so they not only have relatively small computational complexity, but also can carry richer semantic information. However, the traditional GNNs are more rely on the context information of the individual samples and lack geographic prior knowledge that reflects the overall situation of the research area. Therefore, these methods may be disturbed by the confusion of “different objects with the same spectrum” or “violating the first law of geography” in some areas. To address the above problems, we propose a remote sensing semantic segmentation model called knowledge and spatial pyramid distance-based gated graph attention network (KSPGAT), which is based on prior knowledge, spatial pyramid distance and a graph attention network (GAT) with gating mechanism. The model first uses superpixels (geographical objects) to form the nodes of a graph neural network and then uses a novel spatial pyramid distance recognition algorithm to recognize the spatial relationships. Finally, based on the integration of feature similarity and the spatial relationships of geographic objects, a multi-source attention mechanism and gating mechanism are designed to control the process of node aggregation, as a result, the high-level semantics, spatial relationships and prior knowledge can be introduced into a remote sensing semantic segmentation network. The experimental results show that our model improves the overall accuracy by 4.43% compared with the U-Net Network, and 3.80% compared with the baseline GAT network.

show abstract

Cross-Modality Attention with Semantic Graph Embedding for Multi-Label Classification

Cited by 144 publications

References 23 publications

A Serial Attention Frame for Multi-Label Waste Bottle Classification

A Serial Attention Frame for Multi-Label Waste Bottle Classification

Deep tree-ensembles for multi-output prediction

Knowledge and Spatial Pyramid Distance-Based Gated Graph Attention Network for Remote Sensing Semantic Segmentation

Contact Info

Product

Resources

About