HMM word graph based keyword spotting in handwritten document images

Abstract

Line-level keyword spotting (KWS) is presented on the basis of frame-level word posterior probabilities. These posteriors are obtained using word graphs derived from the recognition process of a full-fledged handwritten text recognizer based on hidden Markov models and N-gram language models. This approach has several advantages. First, since it uses a holistic, segmentation-free technology, it does not require any kind of word or character segmentation. Second, the use of language models allows the context of each spotted word to be taken into account, thereby considerably increasing the KWS accuracy. And third, the proposed KWS scores are based on true posterior probabilities, computed taking into account all (or most) possible word segmentations of the input image. These scores are properly bounded and normalized. This mathematically clean formulation lends itself to smooth, threshold-based keyword queries which, in turn, permit comfortable trade-offs between search precision and recall. Experiments are carried out on several historic collections of handwritten text images, as well as with a well-known dataset of modern English handwritten text. According to the empirical results, the proposed approach achieves KWS results comparable to those obtained with the recently-introduced "BLSTM neural networks KWS" approach and clearly outperform the popular, state-of-the-art "Filler HMM" KWS method. Overall, the results clearly support all the above-claimed advantages of the proposed approach.

Word-Graph Based Applications for Handwriting Documents: Impact of Word-Graph Size on Their Performances

“…The goal is to determine whether a given keyword is or is not in each text line image, no matter how many occurrences of the word may appear in the line. According to [2], an adequate global line-level measure S(v, x) to score the degree of presence of a keyword v in a text line (represented by its feature vector sequence x), without considering any specific position within the line image, is given by:…”

“…In [2] it is shown how this probability can be directly and efficiently computed by using WGs. Specifically, it can be obtained by considering the contribution of all the WG edges labelled with v, which correspond to segmentation hypotheses that include the frame i; that is:…”

“…is the forward and β(.) backward accumulated path scores which can be efficiently computed on the WGs by dynamic programming [8,2].…”

“…The AP is based on the standard recall and interpolated precision [10] measures, which are functions of a threshold used to decide whether a score S(v, x) (see (2)) is high enough to assume that a word v is in x. Actually, the AP is defined as the area under the Recall-Precision curve.…”

“…Given the amount of text, automated methods are needed to allow the users to transcribe and/or search, and also to add value to mass-digitisation and preservation efforts of Culture Heritage institutions. To this end, the tranScriptorium 1 project aims at fulfil these requirements with the development of two different applications: the Computer Assisted Transcription for Test Images (CATTI) [1], intended to speed up transcription process, and the KeyWord Spotting [2] for automatic indexing of untranscribed handwritten material under the so called Precision-Recall tradeoff model. Actually, both applications rely on what is called word lattice or Word Graph (WG), which are previouly produced by a standard HMM-based handwritten text recognition system.…”