Insights From A Large-Scale Database of Material Depictions In Paintings

Lin, Hubert; Zuijlen, Mitchell van; Wijntjes, Maarten; Pont, Sylvia C.; Bala, Kavita

doi:10.48550/arxiv.2011.12276

Cited by 3 publications

(3 citation statements)

References 40 publications

(48 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Interesting work has been done on the topic of people and face detection in paintings [122,97,93,123,93], as well as analysis and classification of the detected faces based on gender and other features [118]. Apart from faces, effort has been made to recognize other content-related elements of artworks, such as detecting the pose of characters in paintings [68,86,9], recognizing specific characters [85] or detecting materials depicted in paintings [83].…”

Section: Object Detection and Similarity Retrievalmentioning

confidence: 99%

Understanding and Creating Art with AI: Review and Outlook

Cetinić¹,

She²

2021

Preprint

View full text Add to dashboard Cite

Technologies related to artificial intelligence (AI) have a strong impact on the changes of research and creative practices in visual arts. The growing number of research initiatives and creative applications that emerge in the intersection of AI and art, motivates us to examine and discuss the creative and explorative potentials of AI technologies in the context of art. This paper provides an integrated review of two facets of AI and art: 1) AI is used for art analysis and employed on digitized artwork collections; 2) AI is used for creative purposes and generating novel artworks. In the context of AI-related research for art understanding, we present a comprehensive overview of artwork datasets and recent works that address a variety of tasks such as classification, object detection, similarity retrieval, multimodal representations, computational aesthetics, etc. In relation to the role of AI in creating art, we address various practical and theoretical aspects of AI Art and consolidate related works that deal with those topics in detail. Finally, we provide a concise outlook on the future progression and potential impact of AI technologies on our understanding and creation of art.

show abstract

Section: Object Detection and Similarity Retrievalmentioning

confidence: 99%

Understanding and Creating Art with AI: Review and Outlook

Cetinić¹,

She²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In the last decade, there has been increased interest in exploring computer vision and machine learning methods for analyzing and categorizing digitized collections of paintings. Most of the work in this field is focused on classification [ 5 , 6 , 7 , 8 ], pattern recognition and retrieval [ 9 , 10 , 11 , 12 , 13 ], or object detection in paintings [ 14 , 15 , 16 , 17 ]. Studying bodily depictions in the context of digital art history has so far been mostly exclusively focused on body pose detection and recognition [ 18 , 19 , 20 ].…”

Section: Introductionmentioning

confidence: 99%

A Computational Approach to Hand Pose Recognition in Early Modern Paintings

2023

View full text Add to dashboard Cite

Hands represent an important aspect of pictorial narration but have rarely been addressed as an object of study in art history and digital humanities. Although hand gestures play a significant role in conveying emotions, narratives, and cultural symbolism in the context of visual art, a comprehensive terminology for the classification of depicted hand poses is still lacking. In this article, we present the process of creating a new annotated dataset of pictorial hand poses. The dataset is based on a collection of European early modern paintings, from which hands are extracted using human pose estimation (HPE) methods. The hand images are then manually annotated based on art historical categorization schemes. From this categorization, we introduce a new classification task and perform a series of experiments using different types of features, including our newly introduced 2D hand keypoint features, as well as existing neural network-based features. This classification task represents a new and complex challenge due to the subtle and contextually dependent differences between depicted hands. The presented computational approach to hand pose recognition in paintings represents an initial attempt to tackle this challenge, which could potentially advance the use of HPE methods on paintings, as well as foster new research on the understanding of hand gestures in art.

show abstract

“…They found that it was optimal not to fully retrain a deep residual neural model, concluding that the bottom layers contain filters general enough to be used across datasets. The effect of data preprocessing, feature extraction and fine-tuning of models for classification tasks in the artistic domain is studied in detail in [118,198,227,287]. The study of Cetinic et al (2018) particularly stands out in this regard [54].…”

Section: Computational Approaches To the Artistic Domainmentioning

confidence: 99%

Data-driven Modelling of Cognitive and Affective Variables of Perception of Multimedia Content

Faúndez-Carrasco Población

View full text Add to dashboard Cite

Understanding how external stimuli are transformed into meaningful impressions that guide human actions has been an enduring challenge. Automatic models of perception of multimedia productions stand out as a path to the characterisation of our interaction with a type of content that has nowadays flooded social media platforms and people's digital time. Aided by well-established theories of perception, we identify memorability, attention, judgements and the emotional state as cognitive and affective variables that provide complementary views towards the comprehension of our perception of multimedia content.Intrinsic media memorability is defined as an inherent property of the visual features of videos that determines the percentage of people who remember watching a clip in a second viewing. Our approaches are based on the extraction of video-level, topic-oriented features using pre-trained Transformers. We find that linear models trained using these features as inputs can reach prediction rates comparable to other state-of-the-art models across several datasets. Secondly, we characterise the attention to short movies at a group-level by means of using electrodermal activity recordings as ground-truth. We develop a binary classification system whose predictions, based on a semantically-driven representation of the acoustic signal of videos, denote whether the group-level attention increases or diminishes. Next, we address the judgement of images attending to a fitness criterion to tourism attractiveness. Our proposal builds upon a Mixture-Of-Experts system to leverage information relative to geolocation tags, which implicitly points to specific semantics and contents, seeking to incorporate into the model design the knowledge about the role of context during the annotation process. In order to predict the emotions elicited by historical artworks, we employ vision-language cross-modal models that seek to exploit the subjective and figurative nature within the artistic domain. We introduce a methodology to approximate systems pre-trained in realistic content to the art domain, finding that following it leads to significant improvements (up to 27%) in predicting emotions. However, given the complexity of understanding the rationale behind "black-box" visual prediction models, hence our last contribution targets the enhancement of the interpretability of these systems. We explore how to improve the explanations provided by LIME, a popular surrogate-based, post-hoc explanatory technique, by indirectly adding information about the statistic of the data distribution the "black-box" model is trained on.We believe this thesis contributes to comprehending the human perception of multimedia content by addressing from a computational perspective several cognitive and affective variables that conform to it. In particular, our approaches seek to combine information from multiple modalities, presenting models that extract patterns from low-level features of the inputs and relate them to human actions and responses spanning variou...

show abstract

Insights From A Large-Scale Database of Material Depictions In Paintings

Cited by 3 publications

References 40 publications

Understanding and Creating Art with AI: Review and Outlook

Understanding and Creating Art with AI: Review and Outlook

A Computational Approach to Hand Pose Recognition in Early Modern Paintings

Data-driven Modelling of Cognitive and Affective Variables of Perception of Multimedia Content

Contact Info

Product

Resources

About