Knowledge-driven understanding of images in comic books

Abstract

Document analysis is an active field of research which can attain a complete understanding of the semantics of a given document. One example of the document understanding process is enabling a computer to identify the key elements of a comic book story and arrange them according to a predefined domain knowledge. In this study we propose a knowledge-driven system that can interact with bottomup and top-down information to progressively understand the content of a document. We model the comic book's and the image processing domains knowledge for information consistency analysis. In addition, different image processing methods are improved or developed to extract panels, balloons, tails, texts, comic characters and their semantic relations in an unsupervised way.

Semi-automatic Text and Graphics Extraction of Manga Using Eye Tracking Information

“…We demonstrated that few manually labeled data can make the detection more robust against various postures and facial expression [19]. We recently demonstrate that speech balloon and tail detection allow manga character region of interest extraction and character-to-speech association [20], [21].…”

Digital Comics Image Indexing Based on Deep Learning

“…At such level of precision, it is also possible to localize and find the direction of the tail. We proposed a method for this task in 2015 [5] and also to associate speech balloons and comic characters [35].…”

“…We propose to use our previously published methods for detecting the tail position along the edge of the speech balloons, which determines its direction and associates it with a speaker (comic character) [5,35].…”

“…In [5], tail detection is performed on the analysis of convexity defects of the convex hull of the speech balloon. Once we find the tail position, we compute the orientation and direction of the last part of the tail which is directed towards the speaking character.…”

“…This field of research includes content segmentation [5], relation analysis [6], style analysis [7] and information retrieval [8]. While information retrieval research domain has a long history, content-based comics indexing and the creation of comics in a digital form are quite recent.…”