Understanding Compositional Structures in Art Historical Images Using Pose and Gaze Priors

Madhu, Prathmesh; Marquart, Tilman; Kosti, Ronak; Bell, Peter C.; Maier, Andreas; Christlein, Vincent

doi:10.1007/978-3-030-66096-3_9

Cited by 10 publications

(11 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The latter rely on the iconographic classification system Iconclass, which is conceived for the Western motifs of the visual arts [78]. As a result of the already time-consuming labeling process at image-level, few data sets feature object-level annotations [3,15,29,34,48,64,84,89]. When provided, they are usually marked with bounding boxes, so that object instances are enclosed with rectangles and thus precisely located in the image.…”

Section: Related Workmentioning

confidence: 99%

“…Existing data sets fall broadly into two categories. Either they do index keypoints but are not publicly available and are dedicated to a comparatively narrow subset of art-historical representation practices [34,48]. Or they are freely accessible to the public but enclose human figures only by rectangular bounding boxes; their pose is then broadly categorized without specifically delineating keypoints [64].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Poses of People in Art: A Data Set for Human Pose Estimation in Digital Art History

Schneider,

Vollmer,

Huber

2023

View full text Add to dashboard Cite

Throughout the history of art, the pose-as the holistic abstraction of the human body's expression-has proven to be a constant in numerous studies. However, due to the enormous amount of data that so far had to be processed by hand, its crucial role to the formulaic recapitulation of art-historical motifs since antiquity could only be highlighted selectively. This is true even for the now automated estimation of human poses, as domain-specific, sufficiently large data sets required for training computational models are either not publicly available or not indexed at a fine enough granularity. With the Poses of People in Art data set, we introduce the first openly licensed data set for estimating human poses in art and validating human pose estimators. It consists of 2,454 images from 22 art-historical depiction styles, including those that have increasingly turned away from lifelike representations of the body since the 19 th century. A total of 10,749 human figures are precisely enclosed by rectangular bounding boxes, with a maximum of four per image labeled by up to 17 keypoints; among these are mainly joints such as elbows and knees. For machine learning purposes, the data set is divided into three subsets-training, validation, and testing-, that follow the established JSON-based Microsoft Common Objects in Context (COCO) format, respectively. Each image annotation, in addition to mandatory fields, provides metadata from the art-historical online encyclopedia WikiArt. With this paper, we elaborate on the acquisition and constitution of the data set, address various application scenarios, and discuss prospects for a digitally supported art history. We show that the data set enables the comprehensive investigation of body phenomena in art, whether at the level of individual figures, which can thus be captured in their subtleties, or entire figure constellations, whose position, distance, or proximity to one another is considered.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Poses of People in Art: A Data Set for Human Pose Estimation in Digital Art History

Schneider,

Vollmer,

Huber

2023

View full text Add to dashboard Cite

show abstract

“…A publicly accessible data set that contains poses of human figures in artworks does not yet exist. Relevant previous work employs different approaches to deal with the lack of annotated training data: they (1) analyze only self-annotated data sets, without training models or performing inference [17]; (2) use trained pose estimators from another domain without adaptation [18,29]; (3) apply style transfer to real-world data sets to close the domain gap [30]; or (4) leverage small, keypoint-level annotated data sets to fine-tune pre-trained models [30].…”

Section: Related Workmentioning

confidence: 99%

“…To date, however, only few approaches exist for human pose estimation in art-historical images, possibly due to the lack of a sufficiently large domain-specific data set. To deal with this issue, one type of approaches uses pre-trained models, but without adapting them to the new domain [18,29], while others apply style transfer to real-world data sets to obtain domain-specific training data [30], or fine-tune pre-trained models using small, keypoint-level annotated data sets [30].…”

Section: Introductionmentioning

confidence: 99%

Semi-supervised Human Pose Estimation in Art-historical Images

Springstein¹,

Schneider²,

Althaus³

et al. 2022

Preprint

View full text Add to dashboard Cite

Gesture as 'language' of non-verbal communication has been theoretically established since the 17th century. However, its relevance for the visual arts has been expressed only sporadically. This may be primarily due to the sheer overwhelming amount of data that traditionally had to be processed by hand. With the steady progress of digitization, though, a growing number of historical artifacts have been indexed and made available to the public, creating a need for automatic retrieval of art-historical motifs with similar body constellations or poses. Since the domain of art differs significantly from existing real-world data sets for human pose estimation due to its style variance, this presents new challenges. In this paper, we propose a novel approach to estimate human poses in art-historical images. In contrast to previous work that attempts to bridge the domain gap with pre-trained models or through style transfer, we suggest semi-supervised learning for both object and keypoint detection. Furthermore, we introduce a novel domain-specific art data set that includes both bounding box and keypoint annotations of human figures. Our approach achieves significantly better results than methods that use pre-trained models or style transfer.

show abstract

“…Interesting work has been done on the topic of people and face detection in paintings [122,97,93,123,93], as well as analysis and classification of the detected faces based on gender and other features [118]. Apart from faces, effort has been made to recognize other content-related elements of artworks, such as detecting the pose of characters in paintings [68,86,9], recognizing specific characters [85] or detecting materials depicted in paintings [83].…”

Section: Object Detection and Similarity Retrievalmentioning

confidence: 99%

Understanding and Creating Art with AI: Review and Outlook

Cetinić¹,

She²

2021

Preprint

View full text Add to dashboard Cite

Technologies related to artificial intelligence (AI) have a strong impact on the changes of research and creative practices in visual arts. The growing number of research initiatives and creative applications that emerge in the intersection of AI and art, motivates us to examine and discuss the creative and explorative potentials of AI technologies in the context of art. This paper provides an integrated review of two facets of AI and art: 1) AI is used for art analysis and employed on digitized artwork collections; 2) AI is used for creative purposes and generating novel artworks. In the context of AI-related research for art understanding, we present a comprehensive overview of artwork datasets and recent works that address a variety of tasks such as classification, object detection, similarity retrieval, multimodal representations, computational aesthetics, etc. In relation to the role of AI in creating art, we address various practical and theoretical aspects of AI Art and consolidate related works that deal with those topics in detail. Finally, we provide a concise outlook on the future progression and potential impact of AI technologies on our understanding and creation of art.

show abstract

Understanding Compositional Structures in Art Historical Images Using Pose and Gaze Priors

Cited by 10 publications

References 20 publications

Poses of People in Art: A Data Set for Human Pose Estimation in Digital Art History

Poses of People in Art: A Data Set for Human Pose Estimation in Digital Art History

Semi-supervised Human Pose Estimation in Art-historical Images

Understanding and Creating Art with AI: Review and Outlook

Contact Info

Product

Resources

About