Knowledge Graph Embedding-Based Domain Adaptation for Musical Instrument Recognition

Abstract

Convolutional neural networks raised the bar for machine learning and artificial intelligence applications, mainly due to the abundance of data and computations. However, there is not always enough data for training, especially when it comes to historical collections of cultural heritage where the original artworks have been destroyed or damaged over time. Transfer Learning and domain adaptation techniques are possible solutions to tackle the issue of data scarcity. This article presents a new method for domain adaptation based on Knowledge graph embeddings. Knowledge Graph embedding forms a projection of a knowledge graph into a lower-dimensional where entities and relations are represented into continuous vector spaces. Our method incorporates these semantic vector spaces as a key ingredient to guide the domain adaptation process. We combined knowledge graph embeddings with visual embeddings from the images and trained a neural network with the combined embeddings as anchors using an extension of Fisher’s linear discriminant. We evaluated our approach on two cultural heritage datasets of images containing medieval and renaissance musical instruments. The experimental results showed a significant increase in the baselines and state-of-the-art performance compared with other domain adaptation methods.

AIKoGAM: An AI-driven Knowledge Graph of the Antiquities Market: Toward Automatised Methods to Identify Illicit Trafficking Networks

“…A promising avenue of exploration would involve merging other similar datasets, as the one discussed in Fabiani and Marrone (2021), which encompasses data from external auction catalogues and extend over a wider time frame. Lastly the KG would benefit from the implementation of Computer Vision (CV) and embedding of visual data (Eyharabide, Bekkouch & Constantin 2021). By combining textual and visual data, similarity calculations, clustering and identification of the same objects/highly similar objects described differently and/or with misaligned provenance information would be possible with less human inspection.…”

Few-Shot Object Detection: Application to Medieval Musicological Studies

“…Although the field of object detection is relatively mature and has been around for quite some time, its applications to cultural heritage data have been relatively modest. In the musicology field, most contributions use simple images [28], such as the recent contributions to digital cultural heritage analysis focusing on similarity metric learning methods for making semantic-level judgments, such as predicting a painting's style, genre, and artist [29,30]. Other contributions detect fake artworks through stroke analysis and an artistic style transfer using adversarial networks to regularize the generation of stylized images [31].…”

What augmentations are sensitive to hyper-parameters and why?

“…Augmentation can be performed on different domains, [43,19], an experiment is performed of performing augmentation on text input and image input. The procedures include making a model for generating images based on their domains and applying various affine transformations.…”