PaintTeR: Automatic Extraction of Text Spans for Generating Art-Centered Questions

Abstract

We propose PaintTeR, our Paintings TextRank algorithm for extracting art-related text spans from passages on paintings. PaintTeR combines a lexicon of painting words curated automatically through distant supervision with random walks on a large-scale word co-occurrence graph for ranking passage spans for artistic characteristics. The spans extracted with PaintTeR are used in state-of-the-art Question Generation and Reading Comprehension models for designing an interactive aid that enables gallery and museum visitors focus on the artistic elements of paintings. We provide experiments on two datasets of expert-written passages on paintings to showcase the effectiveness of PaintTeR. Evaluations by both gallery experts as well as crowdworkers indicate that our proposed algorithm can be used to select relevant and interesting art-centered questions. To the best of our knowledge, ours is the first work to effectively fine-tune question generation models using minimal supervision for a low-resource, specialized context such as gallery visits.

Generating Reflective Questions for Engaging Gallery Visitors in ArtMuse

“…A snapshot of Art-Muse in action is shown in Figure 1. 2 Models and Datasets: We used the painting passages from previous work (Gollapalli et al 2022) for generating synthetic sessions for training our RQG model. The chitchat dialog model used in the data augmentation step was trained on the PersonaChat dataset (Zhang et al 2018).…”

“…GLAMs (Galleries, Libraries, Archives, and Museums) present an opportunity for AI applications to bridge the gap between the highly engaging human docents and passive classical audio guides (Schaffer et al 2018;van Strien et al 2022). To this end, several recent works have focused on building quizzes, game-style interfaces, and question answering (QA)-based chatbots using expert-written passages and other metadata of the artefacts for enabling visitor interaction and enhancing their visit experiences (Boiano et al 2018;Ueta et al 2021;Gollapalli et al 2022).…”