Multitask painting categorization by deep multibranch neural network

Abstract

In this work we propose a new deep multibranch neural network to solve the tasks of artist, style, and genre categorization in a multitask formulation. In order to gather clues from low-level texture details and, at the same time, exploit the coarse layout of the painting, the branches of the proposed networks are fed with crops at different resolutions. We propose and compare two different crop strategies: the first one is a random-crop strategy that permits to manage the tradeoff between accuracy and speed; the second one is a smart extractor based on Spatial Transformer Networks trained to extract the most representative subregions. Furthermore, inspired by the results obtained in other domains, we experiment the joint use of hand-crafted features directly computed on the input images along with neural ones.Experiments are performed on a new dataset originally sourced from wikiart.org and hosted by Kaggle, and made suitable for artist, style and genre multitask learning. The dataset here proposed, named MultitaskPainting100k, is composed by 100K paintings, 1508 artists, 125 styles and 41 genres. Our best method, tested on the MultitaskPainting100k dataset, achieves accuracy levels of 56.5%, 57.2%, and 63.6% on the tasks of artist, style and genre prediction respectively.

Compare the performance of the models in art classification

“…[ 20 ] used the RGB and brush stroke information to classify fine-art painting images. [ 9 ] proposed the MultitaskPainting100k dataset and used a spatial transformer network (STN) that was introduced by [ 42 ] with the injection of HOG features to achieve 56.5%, 63.6% and 57.2% success rates in artist, genre and style tasks, respectively. The WikiArt dataset is from the WikiArt website, and as the number of paintings on websites increases over time, there are differences in the selection methods of paintings for different studies, so the number of paintings and the number of categories differ among the algorithms.…”

“…Some of these innovations focus on the way data are imported and the various models used. In [ 9 ], three regions of interest (ROIs) were extracted from the input images using a regions of interest proposal module to identify the important areas of the image. To achieve this goal, this study used a deep multibranch neural network scheme.…”

“…• We adopt three painting datasets to evaluate the classification accuracy of the models. These three datasets are Painting-91 [12], WikiArt-WikiPaintings [13], and MultitaskPainting100k [9], all of which are well-known benchmark datasets for art classification. We use both small and large datasets when testing the various tasks.…”

How to Represent Paintings: A Painting Classification Using Artistic Comments

“…To achieve our goal of classifying paintings using artistic comments, we adopted the SemArt dataset collected from [ 39 ] to classify various metadata, including type, school, timeframe and author. In recent years, several datasets have been used to classify artworks, such as Painting91 [ 40 ], WikiArt-WikiPaintings [ 41 ] and MultitaskPainting100k [ 42 ]; however, these are labeled only by artists, styles and genres. Previous works have focused on the task of identifying paintings based on different artist representations.…”

Painting Practical Support: A Study about the Usage of Painting Materials in Children’s Painting Works

“…Other research fields also research painting materials. The researchers intelligently classify the paintings according to the characteristics of the painting materials in the paintings (Bianco et al 2019). Painting materials can also help people study the age of paintings.…”