Knowledge Graph-Based Image Classification Refinement

Abstract

Biologically inspired ideas are important in image processing. Not only does more than 80% of the information received by humans comes from the visual system, but the human visual system also gives its fast, accurate, and efficient image processing capability. In the current image classification task, convolutional neural networks (CNNs) focus on processing pixels and often ignore the semantic relationships and human brain mechanisms. With the development of image analysis and processing techniques, the information in the image is becoming increasingly complicated. Humans can learn about the characteristics of objects and the relationships that occur between them to classify the images. It is a significant characteristic that sets humans apart from the modern learning-based computer vision algorithms. How to make full use of the semantic relationships in categories and how to apply the knowledge of biological vision to image classification are our main concerns. In this view, we propose the concept of the image knowledge graph (IKG) to incorporate the semantic association and the scene association to fully consider the relations between objects (external and internal). We take full advantage of the reasoning model of the knowledge graph that is closer to the biological visual information-processing model. We conduct extensive experiments on large-scale image datasets (ImageNet), demonstrating the effectiveness of our approach. Furthermore, our method participates in ILSVRC 2017 challenges and obtains the new state-of-the-art results on the ImageNet (82.43%).

GCNBoost: Artwork Classification by Label Propagation through a Knowledge Graph

“…In the recent work on image classification, GCNs are receiving more attention thanks to their ability to model the relationships between a set of entities through a KG, which is effective in the tasks of node classification [28,35] and link prediction [35]. Particularly, GCN are used to model the relations a set of images may have accounting for their labels [6,22,39,43]. The basic idea is to combine the classical visual features with GCN embeddings learned over the KG.…”

“…For example, GCNs are used in [6] to improve multi-labels classification accounting for semantic links between different labels, whereas the authors of [39] improved this idea with WordNet concepts hierarchy, devising a zero-shot classification technique. Finally, the authors in [43] used a weighted adjacency matrix to efficiently model the inter-dependency between image labels. We follow the same idea, introducing pseudo-labeled data, used as true data to train a GCN following a standard label propagation process.…”

CMIM: Cross-Modal Information Maximization For Medical Imaging

“…Multi-modal data has been exploited in numerous medical tasks including: caption generation [3] (text and images), lesion detection [4] (mammogram and MRI), image classification [5] (image and knowledge graphs) and few-shot semantic segmentation [6]. While such systems yield performance improvements, there are few works on creating systems that while benefiting from additional training modalities are robust to modality dropping at test-time.…”

OD-GCN: Object Detection Boosted by Knowledge GCN

“…Xiaolong Wang et al [14] proposed a zero-shot classification network with GCN through a WordNet knowledge graph. D. Zhang et al [22] built a sematic knowledge graph and a scene probability graph, refining the traditional classification results.…”