Can we teach computers to understand art? Domain adaptation for enhancing deep networks capacity to de-abstract art

Abstract

Humans comprehend a natural scene at a single glance; painters and other visual artists, through their abstract representations, stressed this capacity to the limit. The performance of computer vision solutions matched that of humans in many problems of visual recognition. In this paper we address the problem of recognizing the genre (subject) in digitized paintings using Convolutional Neural Networks (CNN) as part of the more general dealing with abstract and/or artistic representation of scenes. Initially we establish the state of the art performance by training a CNN from scratch. In the next level of evaluation, we identify aspects that hinder the CNNs' recognition, such as artistic abstraction. Further, we test various domain adaptation methods that could enhance the subject recognition capabilities of the CNNs. The evaluation is performed on a database of 80,000 annotated digitized paintings, which is tentatively extended with artistic photographs, either original or stylized, in order to emulate artistic representations. Surprisingly, the most efficient domain adaptation is not the neural style transfer. Finally, the paper provides an experiment-based assessment of the abstraction level that CNNs are able to achieve.

Machine-Assisted Learning in Highly-Interdisciplinary Media Fields: A Multimedia Guide on Modern Art

“…The bundle of the implemented TEL services can be utilized for forming innovative education activities, both for in-class tutoring and self-training support, through the use of comprehensive material (e.g., showing items of specific movements, artists, artworks, their temporal evolution/timelines, etc.). Hence, the values of blended learning [26] are extended with the incorporation of representative examples, experiential assignments, simulations, and exercises that can be addressed to individuals or as team-work projects, thus transforming the schooling routine into an augmented interactive experience [30,31,[45][46][47][48][49]. Specifically, photo galleries can be used to highlight the resemblances and dissimilarities of the different species; quizzes can examine the levels of understanding and knowledge, triggering class discussion; puzzles can be exploited to organize personal collections and team-contests [26][27][28].…”

“…Specifically, photo galleries can be used to highlight the resemblances and dissimilarities of the different species; quizzes can examine the levels of understanding and knowledge, triggering class discussion; puzzles can be exploited to organize personal collections and team-contests [26][27][28]. Gaming components can also provide indications of the salient visual parameters that facilitate artwork recognition, which urge trainees to get involved in associated AR-driven TEL services [12,13,[27][28][29][30][31][32][33][34][35][36][37][38][39] (i.e., to identify the parts of the paintings, the revealing of which triggers higher/empirical cognition, thus helping to detect the descriptive features behind the different styles and their classification properties). Feedback queries allow for the harvesting of users' responses (while playing the corresponding movement recognition games), thus receiving useful insights on subjective discrimination difficulties, which could help to better adapt and direct the teaching procedures.…”

“…Extending the above, these processes are also suitable for obtaining useful feedback and crowdsourcing-driven indirect annotation that could further propel artificial art recognition projects [12,13]. Even if these perspectives are considered somewhat beyond the main scope of TEL, the associated efforts are thought as being amongst the most challenging and trendy examples of inter-sectoral research [31][32][33][34][35][36][37][38][39], fueling cross-domain collaboration and knowledge exchange. Overall, the paper attempts to investigate and thoroughly analyze the potential advantages of TEL services in the media business, detecting the difficulties and needs that are associated with the interdisciplinary nature of this area, while stressing out convincing answers and prosperous solutions.…”

“…Returning to the educational perspectives and the offered capabilities, the introduced "learning by example" concept can be inducted in the goals of blended learning, thus giving the opportunity to schedule physical-and/or distant-communication teaching (synchronous or asynchronous). In addition, this new approach is closer to the training of artificial intelligence systems, i.e., through the paradigms of Machine and Deep Learning (ML/DL) [12,13,[31][32][33][34][35][36][37][38][39]. Inspired by this, a painting movement recognition modality was examined (already implied), which offered an alternative/opposite viewpoint, where humans guide the machines.…”

“…Figure 4 presents the envisioned TEL framework behind the conducted work, which included already running services and future perspectives, while also demonstrating the two-folded orientation of the proposed "machine-assisted learning". When considering the intensiveness of present-day research projects that were related to art recognition and attendance [31][32][33]46,47], such labors require interdisciplinary workforce and associative media training that cannot be left out of the scope of this work. Specifically, while mobile devices are not compatible with the offered gaming utilities (as already explained), there was an effort to engage smartphones and tablets in educative exercises on artificial art recognition.…”