Learning Visual Models Using a Knowledge Graph as a Trainer

Monka, Sebastian; Halilaj, Lavdim; Schmid, Stefan; Rettinger, Achim

doi:10.1007/978-3-030-88361-4_21

“…Monka et. al [94] propose KG-NN, an approach that uses a KG and its h s to train a visual DNN. Using a contrastive knowledge graph embedding loss in combination with h s they learn a visual-semantic features extractor f v,s (•).…”

Section: Visual-semantic Features Extractorsmentioning

confidence: 99%

A survey on visual transfer learning using knowledge graphs

Monka

¹

,

Halilaj

²

,

Rettinger

³

2022

Self Cite

View full text Add to dashboard Cite

The information perceived via visual observations of real-world phenomena is unstructured and complex. Computer vision (CV) is the field of research that attempts to make use of that information. Recent approaches of CV utilize deep learning (DL) methods as they perform quite well if training and testing domains follow the same underlying data distribution. However, it has been shown that minor variations in the images that occur when these methods are used in the real world can lead to unpredictable and catastrophic errors. Transfer learning is the area of machine learning that tries to prevent these errors. Especially, approaches that augment image data using auxiliary knowledge encoded in language embeddings or knowledge graphs (KGs) have achieved promising results in recent years. This survey focuses on visual transfer learning approaches using KGs, as we believe that KGs are well suited to store and represent any kind of auxiliary knowledge. KGs can represent auxiliary knowledge either in an underlying graph-structured schema or in a vector-based knowledge graph embedding. Intending to enable the reader to solve visual transfer learning problems with the help of specific KG-DL configurations we start with a description of relevant modeling structures of a KG of various expressions, such as directed labeled graphs, hypergraphs, and hyper-relational graphs. We explain the notion of feature extractor, while specifically referring to visual and semantic features. We provide a broad overview of knowledge graph embedding methods and describe several joint training objectives suitable to combine them with high dimensional visual embeddings. The main section introduces four different categories on how a KG can be combined with a DL pipeline: 1) Knowledge Graph as a Reviewer; 2) Knowledge Graph as a Trainee; 3) Knowledge Graph as a Trainer; and 4) Knowledge Graph as a Peer. To help researchers find meaningful evaluation benchmarks, we provide an overview of generic KGs and a set of image processing datasets and benchmarks that include various types of auxiliary knowledge. Last, we summarize related surveys and give an outlook about challenges and open issues for future research.

show abstract

“…Monka et. al [95] propose KG-NN, an approach that uses a KG and its h s to train a visual DNN. Using a contrastive knowledge graph embedding loss in combination with h s they learn a visual-semantic features extractor f v,s (•).…”

Section: Visual-semantic Features Extractorsmentioning

confidence: 99%

A Survey on Visual Transfer Learning using Knowledge Graphs

Monka¹,

Halilaj²,

Rettinger³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

The information perceived via visual observations of real-world phenomena is unstructured and complex. Computer vision (CV) is the field of research that attempts to make use of that information. Recent approaches of CV utilize deep learning (DL) methods as they perform quite well if training and testing domains follow the same underlying data distribution. However, it has been shown that minor variations in the images that occur when these methods are used in the real world can lead to unpredictable and catastrophic errors. Transfer learning is the area of machine learning that tries to prevent these errors. Especially, approaches that augment image data using auxiliary knowledge encoded in language embeddings or knowledge graphs (KGs) have achieved promising results in recent years. This survey focuses on visual transfer learning approaches using KGs, as we believe that KGs are well suited to store and represent any kind of auxiliary knowledge. KGs can represent auxiliary knowledge either in an underlying graph-structured schema or in a vector-based knowledge graph embedding. Intending to enable the reader to solve visual transfer learning problems with the help of specific KG-DL configurations we start with a description of relevant modeling structures of a KG of various expressions, such as directed labeled graphs, hypergraphs, and hyper-relational graphs. We explain the notion of feature extractor, while specifically referring to visual and semantic features. We provide a broad overview of knowledge graph embedding methods and describe several joint training objectives suitable to combine them with high dimensional visual embeddings. The main section introduces four different categories on how a KG can be combined with a DL pipeline: 1) Knowledge Graph as a Reviewer; 2) Knowledge Graph as a Trainee; 3) Knowledge Graph as a Trainer; and 4) Knowledge Graph as a Peer. To help researchers find meaningful evaluation benchmarks, we provide an overview of generic KGs and a set of image processing datasets and benchmarks that include various types of auxiliary knowledge. Last, we summarize related surveys and give an outlook about challenges and open issues for future research.

show abstract

“…• WordNet is the most widely used KG for augmenting both ZSL [2,5,36,62,63,88,98,103,105,183,188,192] and FSL [2,39,85,125,139,174]. As a large lexical database with several different relationships between words, such as synonym, hyponym, hypernym and meronym [122], it is often used to build task-specific class hierarchies, especially for image classification.…”

Section: Sub-kg Extractionmentioning

confidence: 99%

“…In extracting neighbours of the matched entities, when more hops considered, the extracted sub-KG has a higher knowledge coverage but more irrelevant knowledge. Besides the k-hops neighbourhood, some studies such as [2] and [125] just extract the ancestors or parents of the matched nodes for simple class hierarchies as a sub-KG.…”

Section: Sub-kg Extractionmentioning

confidence: 99%

See 1 more Smart Citation

Zero-shot and Few-shot Learning with Knowledge Graphs: A Comprehensive Survey

Chen¹,

Geng²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Machine learning methods especially deep neural networks have achieved great success but many of them often rely on a number of labeled samples for training. In real-world applications, we often need to address sample shortage due to e.g., dynamic contexts with emerging prediction targets and costly sample annotation. Therefore, low-resource learning, which aims to learn robust prediction models with no enough resources (especially training samples), is now being widely investigated. Among all the low-resource learning studies, many prefer to utilize some auxiliary information in the form of Knowledge Graph (KG), which is becoming more and more popular for knowledge representation, to reduce the reliance on labeled samples. In this survey, we very comprehensively reviewed over 90 papers about KG-aware research for two major low-resource learning settings -zero-shot learning (ZSL) where new classes for prediction have never appeared in training, and few-shot learning (FSL) where new classes for prediction have only a small number of labeled samples that are available. We first introduced the KGs used in ZSL and FSL studies as well as the existing and potential KG construction solutions, and then systematically categorized and summarized KG-aware ZSL and FSL methods, dividing them into different paradigms such as the mapping-based, the data augmentation, the propagation-based and the optimization-based. We next presented different applications, including not only KG augmented prediction tasks in Computer Vision and Natural Language Processing (e.g., image classification, visual question answering, text classification and knowledge extraction), but also tasks for KG curation (e.g., inductive KG completion), and some typical evaluation resources for each task. We eventually discussed some challenges and future directions on aspects such as new learning and reasoning paradigms, and the construction of high quality KGs.

show abstract

Learning Visual Models Using a Knowledge Graph as a Trainer

Cited by 14 publications

References 28 publications

A survey on visual transfer learning using knowledge graphs

A survey on visual transfer learning using knowledge graphs

A Survey on Visual Transfer Learning using Knowledge Graphs

Zero-shot and Few-shot Learning with Knowledge Graphs: A Comprehensive Survey

Contact Info

Product

Resources

About