A limited-size ensemble of homogeneous CNN/LSTMs for high-performance word classification

Abstract

The strength of long short-term memory neural networks (LSTMs) that have been applied is more located in handling sequences of variable length than in handling geometric variability of the image patterns. In this paper, an end-to-end convolutional LSTM neural network is used to handle both geometric variation and sequence variability. The best results for LSTMs are often based on large-scale training of an ensemble of network instances. We show that high performances can be reached on a common benchmark set by using proper data augmentation for just five such networks using a proper coding scheme and a proper voting scheme. The networks have similar architectures (convolutional neural network (CNN): five layers, bidirectional LSTM (BiLSTM): three layers followed by a connectionist temporal classification (CTC) processing step). The approach assumes differently scaled input images and different feature map sizes. Three datasets are used: the standard benchmark RIMES dataset (French); a historical handwritten dataset KdK (Dutch); the standard benchmark George Washington (GW) dataset (English). Final performance obtained for the word-recognition test of RIMES was 96.6%, a clear improvement over other state-of-the-art approaches which did not use a pre-trained network. On the KdK and GW datasets, our approach also shows good results. The proposed approach is deployed in the Monk search engine for historical-handwriting collections.

Improving the robustness of LSTMs for word classification using stressed word endings in dual-state word-beam search

“…The network is summarized in Table II. This setup showed a good performance in the earlier work [18]. Moreover, the ensemble does not need handcrafted features or extensive network-architecture engineering efforts.…”

“…In this section, first, we explain four label-coding schemes. Then, we use the ensemble of five CNN-LSTMs suggested in [18], for acknowledging their effect. Such networks can handle sequence variability and geometric variation.…”

“…Hence, in this paper, we explore whether a stressed word start and ending in the ground truth helps the neural network. Using stressed word-start and ending signs results in four possible label-coding schemes (Table I): 1) W The Plain label-coding scheme: The corresponding ground-truth label exclusively shows the characters which are present in the word image, [18]. 2)ˆW The stressed word-start label-coding scheme: An extra character (e.g., 'ˆ') is added to the ground-truth label of a word image to provide the recognizer an extra hint relating the start-of-word shape condition.…”

“…We use an ensemble framework consisting of end-toend trained networks [18]. This approach consists of five individual CNN-LSTM networks and one voter module.…”

“…Apart from the number of units in the hidden layers, important hyperparameter values are the same in the classifiers of the ensemble. Table III shows the number of hidden layers, as in [18]. The batch size is 50.…”