End-to-End Measure for Text Recognition

Abstract

Measuring the performance of text recognition and text line detection engines is an important step to objectively compare systems and their configuration. There exist wellestablished measures for both tasks separately. However, there is no sophisticated evaluation scheme to measure the quality of a combined text line detection and text recognition system. The F-measure on word level is a well-known methodology, which is sometimes used in this context. Nevertheless, it does not take into account the alignment of hypothesis and ground truth text and can lead to deceptive results. Since users of automatic information retrieval pipelines in the context of text recognition are mainly interested in the end-to-end performance of a given system, there is a strong need for such a measure. Hence, we present a measure to evaluate the quality of an end-to-end text recognition system. The basis for this measure is the well established and widely used character error rate, which is limited -in its original form -to aligned hypothesis and ground truth texts. The proposed measure is flexible in a way that it can be configured to penalize different reading orders between the hypothesis and ground truth and can take into account the geometric position of the text lines. Additionally, it can ignore over-and under-segmentation of text lines. With these parameters it is possible to get a measure fitting best to its own needs.Index Terms-measure, end-to-end, character error rate, word error rate, F-measure, bag-of-word, HTR

End-to-End Page-Level Assessment of Handwritten Text Recognition

“…The RO provided by reference transcripts and/or other layout GT annotations is generally only one among several possible RO annotations which would be all correct. Therefore, mixing RO and word recognition errors into a single assessment measure (as in [14,25]) does mot seem the best idea for understanding which are the inner issues of an end-to-end full-page HTR system.…”

“…Finally, both measures are somehow combined to obtain a single scalar figure which hopefully represents an "overall performance" metric [14]. In a similar vein, but explicitly devoted to HTR evaluation, the work presented in [25],goes deeper in the metric combination idea, with daunting mathematical formulation. However, this is a utterly theoretical work which does not provide any empirical evidence that would support the proposed formulation or methods in practice.…”

“…The difficulties underlying the evaluation of page-level HTR results boil down to a Reading Order (RO) problem [5,30,11,41,36,47]. A number of recent proposals try to heuristically weight and combine both word recognition and LA geometric errors into a single scalar value [25,14]. Unfortunately, this hinders the capability to sort out the nature of the corresponding errors and thereby making a comprehensive, useful assessment.…”

A survey of OCR evaluation tools and metrics

“…CER is the inverted accuracy, and defined as CER = (i + s + d) / n, where n is the total number of characters, i the minimal number of character insertions, s the substitutions and d the deletions required to transform the reference text into the OCR output. 6 [16] propose an "end-to-end measure" which is based on the CER, but with alignment between GT and OCR results in a way that makes it configurable whether differences in the reading order or the over-/under-segmentation of text lines are penalized.…”

End-to-End Page-Level Assessment Of Handwritten Text Recognition

“…The difficulties underlying the evaluation of page-level HTR results boil down to a Reading Order (RO) problem [7,26,30,33] . A number of recent proposals try to heuristically weight and combine both word recognition and LA geometric errors into a single scalar value [10,19] . Unfortunately, this hinders the capability to sort out the nature of the corresponding errors and thereby making a comprehensive, useful assessment.…”