Leveraging Known Data for Missing Label Prediction in Cultural Heritage Context

Abstract

Cultural heritage represents a reliable medium for history and knowledge transfer. Cultural heritage assets are often exhibited in museums and heritage sites all over the world. However, many assets are poorly labeled, which decreases their historical value. If an asset’s history is lost, its historical value is also lost. The classification and annotation of overlooked or incomplete cultural assets increase their historical value and allows the discovery of various types of historical links. In this paper, we tackle the challenge of automatically classifying and annotating cultural heritage assets using their visual features as well as the metadata available at hand. Traditional approaches mainly rely only on image data and machine-learning-based techniques to predict missing labels. Often, visual data are not the only information available at hand. In this paper, we present a novel multimodal classification approach for cultural heritage assets that relies on a multitask neural network where a convolutional neural network (CNN) is designed for visual feature learning and a regular neural network is used for textual feature learning. These networks are merged and trained using a shared loss. The combined networks rely on both image and textual features to achieve better asset classification. Initial tests related to painting assets showed that our approach performs better than traditional CNNs that only rely on images as input.

Architectural Heritage Recognition in Historical Film Footage Using Neural Networks

“…However only few studies in Cultural Heritage has grown up in this area. To date, using this approach, researchers have been able to: classify elements of interest in images of buildings with an architectural heritage value (Llamas et al, 2017); detect different monuments based on the features of the monument's images (Saini et al, 2017); automatically annotate Cultural Heritage assets using their visual features as well as the metadata available at hand (Belhi et al, 2018); and develop a mobile app to recognize monuments (Palma, 2019). Besides the improvements of Machine Learning techniques, hardware development, in particular the use of Graphical Processing Units (GPUs), has given boost to the computational efficiency of such algorithms.…”

A Mobile App to Learn About Cultural and Historical Associations in a Closed Loop with Humanities Experts

“…The CrossCult project has pursued the creation of an open technological platform for web/mobile applications to deliver historical and cultural contents, driven by cross-border associations and crosscutting topics. By harnessing recent advances in the areas of knowledge modelling for cultural heritage [6,7], personalization [8][9][10], visualization [11,12] and digital storytelling [13,14], the ultimate goal was to foster new ways for the citizens of EU countries and neighboring ones to appraise their past, understanding common traits and facts while embracing their potentially diverging viewpoints.…”

Cultural Heritage Design Element Labeling System With Gamification

“…Cultural heritage (CH) artifacts are an important link between cultures, as well as between the past and the future [1]. These objects have significant historical and educational value; shapes, decorations, and materials give us insight into the beliefs, economic trends, and lifestyles of the people in a particular region or time period [2], [3], [4].…”