Attribute CNNs for word spotting in handwritten documents

Abstract

Word spotting has become a field of strong research interest in document image analysis over the last years. Recently, AttributeSVMs were proposed which predict a binary attribute representation [3]. At their time, this influential method defined the state-of-theart in segmentation-based word spotting. In this work, we present an approach for learning attribute representations with Convolutional Neural Networks (CNNs). By taking a probabilistic perspective on training CNNs, we derive two different loss functions for binary and real-valued word string embeddings. In addition, we propose two different CNN architectures, specifically designed for word spotting. These architectures are able to be trained in an end-to-end fashion. In a number of experiments, we investigate the influence of different word string embeddings and optimization strategies. We show our Attribute CNNs to achieve state-of-the-art results for segmentation-based word spotting on a large variety of data sets.

HWNet v2: an efficient word image representation for handwritten documents

“…Taking inspirations for word attributes, for handwritten images, Poznanski et al [52] adapted VGGNet [72] for recognizing phoc attributes by having multiple parallel fully connected layers, each one predicting phoc attributes at a particular level. In similar spirits, different architectures [35,74,76,83] were proposed using cnn networks which embed features into different textual embedding spaces defined by phoc. In [74], Sudholt et al proposes an architecture to directly embed image features to phoc attributes by having sigmoid activation in the final layer and thereby avoiding multiple fully connected layers as presented in [52].…”

“…The next set of methods in this space use the principle of attribute embedding framework using deep cnn networks. Here, PHOCNet [74] and TPP-PHOCNet [75,76] uses the output space of cnn as phoc embedding while Triplet-CNN [83] explores with different embeddings such as phoc, dctow and few semantic embeddings. In the table, we report the best performance of Triplet-CNN across different proposed embeddings.…”

“…When the data is limited, fine-tuning a pre-trained network has also been demonstrated to be very effective. In the domain of document images these features have shown better performance for word spotting [37,74,76,83], recognition [52], document classification [26], layout analysis [10], etc. In this work, we propose a deep cnn architecture named as HWNet v2, for the task of learning an efficient word level representation for handwritten documents which can handle multiple writers and, is robust to common forms of degradation and noise.…”

Exploring Confidence Measures for Word Spotting in Heterogeneous Datasets

“…e.g. [2], [3]) require a previous segmentation of document pages into individual word images, which is in general not an easy to solve problem. The segmentation-free approach does not pose this requirement, but aims at solving the retrieval and segmentation problem jointly.…”

“…Our baseline word spotting system is based on the design of [2]. The attribute embedding is a 4-level PHOC representation of partitions 1, 2, 4 and 8 based on the lower case Latin alphabet plus digits.…”