A Computational Approach to Hand Pose Recognition in Early Modern Paintings

Abstract

Hands represent an important aspect of pictorial narration but have rarely been addressed as an object of study in art history and digital humanities. Although hand gestures play a significant role in conveying emotions, narratives, and cultural symbolism in the context of visual art, a comprehensive terminology for the classification of depicted hand poses is still lacking. In this article, we present the process of creating a new annotated dataset of pictorial hand poses. The dataset is based on a collection of European early modern paintings, from which hands are extracted using human pose estimation (HPE) methods. The hand images are then manually annotated based on art historical categorization schemes. From this categorization, we introduce a new classification task and perform a series of experiments using different types of features, including our newly introduced 2D hand keypoint features, as well as existing neural network-based features. This classification task represents a new and complex challenge due to the subtle and contextually dependent differences between depicted hands. The presented computational approach to hand pose recognition in paintings represents an initial attempt to tackle this challenge, which could potentially advance the use of HPE methods on paintings, as well as foster new research on the understanding of hand gestures in art.

Chiroscript: Transcription System for Studying Hand Gestures in Early Modern Painting

“…An interesting perspective would be the combination of our method with technologies for gesture recognition in art. In a recent study, a computational method of recognition of nine canonical chirograms was established and resulted in the creation of a dataset with around 1400 correctly identified and annotated hand images (Bernasconi et al 2023). Based on 2D hand keypoint features and HPE (human pose mation), this technology fills an important gap in the study of chirograms way for new measures and enquiries.…”