A survey on visual transfer learning using knowledge graphs

Abstract

The information perceived via visual observations of real-world phenomena is unstructured and complex. Computer vision (CV) is the field of research that attempts to make use of that information. Recent approaches of CV utilize deep learning (DL) methods as they perform quite well if training and testing domains follow the same underlying data distribution. However, it has been shown that minor variations in the images that occur when these methods are used in the real world can lead to unpredictable and catastrophic errors. Transfer learning is the area of machine learning that tries to prevent these errors. Especially, approaches that augment image data using auxiliary knowledge encoded in language embeddings or knowledge graphs (KGs) have achieved promising results in recent years. This survey focuses on visual transfer learning approaches using KGs, as we believe that KGs are well suited to store and represent any kind of auxiliary knowledge. KGs can represent auxiliary knowledge either in an underlying graph-structured schema or in a vector-based knowledge graph embedding. Intending to enable the reader to solve visual transfer learning problems with the help of specific KG-DL configurations we start with a description of relevant modeling structures of a KG of various expressions, such as directed labeled graphs, hypergraphs, and hyper-relational graphs. We explain the notion of feature extractor, while specifically referring to visual and semantic features. We provide a broad overview of knowledge graph embedding methods and describe several joint training objectives suitable to combine them with high dimensional visual embeddings. The main section introduces four different categories on how a KG can be combined with a DL pipeline: 1) Knowledge Graph as a Reviewer; 2) Knowledge Graph as a Trainee; 3) Knowledge Graph as a Trainer; and 4) Knowledge Graph as a Peer. To help researchers find meaningful evaluation benchmarks, we provide an overview of generic KGs and a set of image processing datasets and benchmarks that include various types of auxiliary knowledge. Last, we summarize related surveys and give an outlook about challenges and open issues for future research.

Editorial of the Special Issue on Deep Learning and Knowledge Graphs

“…The second paper, "A Survey on Visual Transfer Learning using Knowledge Graphs" [11], by Sebastian Monka, Lavdim Halilaj, and Achim Rettinger is a comprehensive analysis of how the rising field of transfer learning is taking advantage of KGs. Specifically, KGs are typically used for representing auxiliary knowledge either in an underlying graph-structured schema or in a vector-based KG embedding.…”

Context-Driven Visual Object Recognition Based on Knowledge Graphs

“…Intuitively, a different context leads to a different representation in the vector space, where h KGE view reflects all relationships that are modelled in GKG view . As illustrated in Figure 4, we present two different ways of learning a visual context embedding h v(GKG view ) following Monka et al [33]. The first one is DN N KGE view u , which uses the knowledge graph as a trainer [34]…”

A Survey on Knowledge Graph-based Methods for Automated Driving

“…More recently, ontologies and KGs have gained interest for knowledge-infused learning approaches. Monka et al [75] provided a survey about visual transfer learning using KGs. However, we did not find a survey that cover the use of KGs applied to AD.…”