ContextNet: representation and exploration for painting classification and retrieval in context

García, Noa; Renoust, Benjamin; Nakashima, Yuta

doi:10.1007/s13735-019-00189-4

Cited by 30 publications

(34 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Instead of aiming to retrieve the images that are most similar to a query image, crossmodal retrieval aims at finding the images most closely related to a provided query text or at finding the best descriptive texts for a query image. Cross-modal image retrieval plays an important role in the context of querying art collections, e.g., [18,19], where it is a challenging task to match images and texts in cultural heritage related collections [48]. In [18], descriptors are learned by minimizing a variant of the triplet loss, where image descriptors and text descriptors are forced to be similar with respect to their dot product.…”

Section: Image Retrieval For Cultural Heritagementioning

confidence: 99%

“…In [18], descriptors are learned by minimizing a variant of the triplet loss, where image descriptors and text descriptors are forced to be similar with respect to their dot product. The approach in [19] also addresses cross-modal retrieval using strategies that are similar to the ones used in our work. The authors obtain image descriptors for retrieval on the basis of a CNN (ContextNet) pre-trained for the multi-task classification of four semantic variables.…”

Section: Image Retrieval For Cultural Heritagementioning

confidence: 99%

“…To solve this problem, it is desirable to generate the training samples automatically by defining similarity based on additional information that is assigned to images; e.g., class labels describing the type of the depicted object [11][12][13][14] or descriptive texts [15,16]. This strategy for generating training data was also applied for image retrieval in the context of digital collections of works of art [17][18][19]. It allows the generation of samples consisting of pairs of images with a known similarity status from existing datasets containing images with annotations.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Deep Descriptor Learning with Auxiliary Classification Loss for Retrieving Images of Silk Fabrics in the Context of Preserving European Silk Heritage

Dorozynski

Rottensteiner

2022

IJGI

View full text Add to dashboard Cite

With the growing number of digitally available collections consisting of images depicting relevant objects from the past in relation with descriptive annotations, the need for suitable information retrieval techniques is becoming increasingly important to support historians in their work. In this context, we address the problem of image retrieval for searching records in a database of silk fabrics. The descriptors, used as an index to the database, are learned by a convolutional neural network, exploiting the available annotations to automatically generate training data. Descriptor learning is combined with auxiliary classification loss with the aim of supporting the clustering in the descriptor space with respect to the properties of the depicted silk objects, such as the place or time of origin. We evaluate our approach on a dataset of fabric images in a kNN-classification, showing promising results with respect to the ability of the descriptors to represent semantic properties of silk fabrics; integrating the auxiliary loss improves the overall accuracy by 2.7% and the average F1 score by 5.6%. It can be observed that the largest improvements can be obtained for variables with imbalanced class distributions. An evaluation on the WikiArt dataset demonstrates the transferability of our approach to other digital collections.

show abstract

Section: Image Retrieval For Cultural Heritagementioning

confidence: 99%

Section: Image Retrieval For Cultural Heritagementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep Descriptor Learning with Auxiliary Classification Loss for Retrieving Images of Silk Fabrics in the Context of Preserving European Silk Heritage

Dorozynski

Rottensteiner

2022

IJGI

View full text Add to dashboard Cite

show abstract

“…One of the most important concepts related to DL is transfer learning [1,[18][19][20][21][22][23][24][25][26][27][28][29][30]. Popular programming and software development platforms such as Matlab or Python offer a wide range of pre-trained CNN models of different structures and complexity.…”

Section: Related Workmentioning

confidence: 99%

Adversarial Learning Approach to Unsupervised Labeling of Fine Art Paintings

2021

View full text Add to dashboard Cite

An automatic classification of fine art images is limited by the scarcity of high-quality labels made by art experts. This study aims to provide meaningful automatic labeling of fine art paintings (machine labeling) without the need for human annotation. A new unsupervised Adversarial Clustering System (ACS) is proposed. The ACS is an adversarial learning approach comprising an unsupervised clustering module generating machine labels and a supervised classification module classifying the data based on the machine labels. Both modules are linked through an optimization algorithm iteratively improving the unsupervised clusters. The objective function driving the improvement consists of the within-cluster sum of squares (WCSS) error and the supervised classification accuracy. The proposed method was tested on three different fine-art datasets, including two sets of paintings previously categorized by art experts and one never categorized collection of Australian Aboriginal paintings. The unsupervised clusters were analyzed using standard unsupervised clustering metrics and a reliability measure between machine and human labeling. The ACS showed higher reliability compared to the classical k-means clustering method. The content analysis of unsupervised clusters indicated grouping based on scene composition, type, and shape of the object, edge sharpness and direction, and color palette.

show abstract

“…Besides finding objects and classifying paintings, art historians are interested in visual similarities and patterns between artworks to gain new knowledge about artistic relations or the meaning and development of motifs. To solve this task without manual effort, researchers proposed algorithms to find visual similarities in art collections automatically [ 17 , 45 , 46 ]. While these approaches show promising results, they are limited to small collections since they require a pairwise comparison of all images and thus, scale quadratically with the dataset size.…”

Section: Related Workmentioning

confidence: 99%

Large-scale interactive retrieval in art collections using multi-style feature aggregation

et al. 2021

View full text Add to dashboard Cite

Finding objects and motifs across artworks is of great importance for art history as it helps to understand individual works and analyze relations between them. The advent of digitization has produced extensive digital art collections with many research opportunities. However, manual approaches are inadequate to handle this amount of data, and it requires appropriate computer-based methods to analyze them. This article presents a visual search algorithm and user interface to support art historians to find objects and motifs in extensive datasets. Artistic image collections are subject to significant domain shifts induced by large variations in styles, artistic media, and materials. This poses new challenges to most computer vision models which are trained on photographs. To alleviate this problem, we introduce a multi-style feature aggregation that projects images into the same distribution, leading to more accurate and style-invariant search results. Our retrieval system is based on a voting procedure combined with fast nearest-neighbor search and enables finding and localizing motifs within an extensive image collection in seconds. The presented approach significantly improves the state-of-the-art in terms of accuracy and search time on various datasets and applies to large and inhomogeneous collections. In addition to the search algorithm, we introduce a user interface that allows art historians to apply our algorithm in practice. The interface enables users to search for single regions, multiple regions regarding different connection types and holds an interactive feedback system to improve retrieval results further. With our methodological contribution and easy-to-use user interface, this work manifests further progress towards a computer-based analysis of visual art.

show abstract

ContextNet: representation and exploration for painting classification and retrieval in context

Cited by 30 publications

References 49 publications

Deep Descriptor Learning with Auxiliary Classification Loss for Retrieving Images of Silk Fabrics in the Context of Preserving European Silk Heritage

Deep Descriptor Learning with Auxiliary Classification Loss for Retrieving Images of Silk Fabrics in the Context of Preserving European Silk Heritage

Adversarial Learning Approach to Unsupervised Labeling of Fine Art Paintings

Large-scale interactive retrieval in art collections using multi-style feature aggregation

Contact Info

Product

Resources

About