Advancing the state of the art for handwritten math recognition: the CROHME competitions, 2011–2014

Abstract

International audienceThe CROHME competitions have helped organize the field of handwritten mathematical expression recognition. This paper presents the evolution of the competition over its first 4 years, and its contributions to handwritten math recognition, and more generally structural pattern recognition research. The competition protocol, evaluation metrics and datasets are presented in detail. Participating systems are analyzed and compared in terms of the central mathematical expression recognition tasks: (1) symbol segmentation, (2) classification of individual symbols, (3) symbol relationships and (4) structural analysis (parsing). The competition led to the development of label graphs, which allow recognition results with conflicting segmentations to be directly compared and quantified using Hamming distances. We introduce structure confusion histograms that provide frequencies for incorrect subgraphs corresponding to ground-truth label subgraphs of a given size and present structure confusion histograms for symbol bigrams (two symbols with a relationship) for CROHME 2014 systems. We provide a novel analysis combining results from competing systems at the level of individual strokes and stroke pairs; this virtual merging of system outputs allows us to more closely examine limitations for current state-of-the-art systems. Datasets along with evaluation and visualization tools produced for the competition are publicly available

Multi-modal Attention Network for Handwritten Mathematical Expression Recognition

“…The main metric in this study is expression recognition rate (ExpRate) [49], i.e., the percentage of predicted mathematical expressions matching the ground truth. Besides, we list the structure recognition rate (StruRate) [49], which only focuses on whether the structure is correctly recognized and ignores symbol recognition errors. In this section, we examine the effectiveness of single-modal SCAN.…”

Online flowchart understanding by combining max-margin Markov random field with grammatical analysis

“…Significant research effort has been reported for recognition of math expressions in the recent years [19], in part due to online input devices becoming more popular. Zanibbi et al [34] proposed baseline structure tree that parses the elements of a mathematical expression into a tree that fits natively its structure and then passed it to lexical parser and expression analyzer.…”

“…Consequently, we can evaluate structural correctness in our graph level recognition analysis. Our graphical level evaluation is done using the tools mentioned in [19]. Table 4 gives object level recognition result, specifically symbol segmentation results and symbol recognition results.…”

“…From Figure 1 we can see that arrows in a flowchart can have various shapes, and can point to almost everywhere. Flowchart also cover a larger scope than mathematical expressions [9,19]. All these add difficulties in sketched diagram recognition.…”

“…Although most of the 2D handwritten languages can be interpreted as the trilogy that has been mentioned above, recent researches tend to use a global method to avoid the propagation of errors, which decrease the recognition rate by feeding the misinterpreted symbols or structures into the following steps [9,19]. Instead of enlarging the search space by generating several segmentations and several symbol hypotheses, our proposal is to guide the local recognition using neighborhood information.…”