Materials In Paintings (MIP): An interdisciplinary dataset for perception, art history, and computer vision

Abstract

In this paper, we capture and explore the painterly depictions of materials to enable the study of depiction and perception of materials through the artists’ eye. We annotated a dataset of 19k paintings with 200k+ bounding boxes from which polygon segments were automatically extracted. Each bounding box was assigned a coarse material label (e.g., fabric) and half was also assigned a fine-grained label (e.g., velvety, silky). The dataset in its entirety is available for browsing and downloading at materialsinpaintings.tudelft.nl. We demonstrate the cross-disciplinary utility of our dataset by presenting novel findings across human perception, art history and, computer vision. Our experiments include a demonstration of how painters create convincing depictions using a stylized approach. We further provide an analysis of the spatial and probabilistic distributions of materials depicted in paintings, in which we for example show that strong patterns exists for material presence and location. Furthermore, we demonstrate how paintings could be used to build more robust computer vision classifiers by learning a more perceptually relevant feature representation. Additionally, we demonstrate that training classifiers on paintings could be used to uncover hidden perceptual cues by visualizing the features used by the classifiers. We conclude that our dataset of painterly material depictions is a rich source for gaining insights into the depiction and perception of materials across multiple disciplines and hope that the release of this dataset will drive multidisciplinary research.

The Legacy of Willem Beurs ‒ Bridging the Gap between Art and Material Perception

“…This was demonstrated by applying quantitative image analysis to the highlights, by measuring and correlating the orientations of ellipses fitted onto the highlights and the grapes, for different levels of perceived glossiness. Van Zuijlen et al (2021) performed an annotation experiment of highlights depicted on drinking glasses using images of paintings and found a consistent stylized pattern resulting in the convincing rendering of glass. When using only computer-rendered stimuli it is more difficult, if not impossible, to reveal hidden information because all the features available in the stimulus are carefully tuned and controlled by the input parameters during the rendering.…”

“…Likewise, according to the modern theory of vision science that regards inverse optics as ill-posed and unfeasible (Fleming, 2017;Nishida, 2019;Purves et al, 2011), the brain has no need (and no way) to compute the physical parameters of a Bidirectional Reflectance Distribution Function (BRDF) (Nicodemus, 1965) to estimate the reflectance of that same glass. Van Zuijlen et al (2021) reported an example of such perception-based, standard material depiction. They found that the rendering of wine glasses in paintings, like the one in Fig.…”

Unsupervised learning reveals interpretable latent representations for translucency perception

“…This problem is especially amplified in translucency (see [9,37] for reviews). Recently, data-driven approaches have attempted to learn material representations by capturing the statistical structure of material appearance across many image samples [10,32,35,[73][74][75][76][77][78]. These approaches have been used to model human perception.…”

What do they make us see: a comparative study of cultural bias in online databases of two large museums

“…The first is the museum's API dataset, which can be accessed via an API or downloaded to a personal computer, enabling users to search and analyze the information in the database directly. This type of database is mainly intended for commercial and academic parties that can study and research the information in it or use it to improve computational models (Tallon, 2018;Van Zuijlen et al, 2021;Villaespesa and Crider, 2021). A second type of database is accessible to end users through the museum's online search system, i.e.…”