Recognition and retrieval of mathematical expressions

Abstract

Document recognition and retrieval technologies complement one another, providing improved access to increasingly large document collections. While recognition and retrieval of textual information is fairly mature, with wide-spread availability of Optical Character Recognition (OCR) and text-based search engines, recognition and retrieval of graphics such as images, figures, tables, diagrams, and mathematical expressions are in comparatively early stages of research. This paper surveys the state of the art in recognition and retrieval of mathematical expressions, organized around four key problems in math retrieval (query construction, normalization, indexing, and relevance feedback), and four key problems in math recognition (detecting expressions, detecting and classifying symbols, analyzing symbol layout, and constructing a representation of meaning). Of special interest is the machine learning problem of jointly optimizing the component algorithms in a math recognition system, and developing effective indexing, retrieval and relevance feedback algorithms for math retrieval. Another important open problem is developing user interfaces that seamlessly integrate recognition and retrieval. Activity in these important research areas is increasing, in part because math notation provides an excellent domain for studying problems common to many document and graphics recognition and retrieval applications, and also because mature applications will likely provide substantial benefits for education, research, and mathematical literacy.

Multimodal Mathematical Expressions Recognition: Case of Speech and Handwriting

“…This is mainly done at two levels: symbol identification (segmentation and recognition) and relationships discovering (through which the identified symbols are spatially arranged). Thus, recognizing a handwritten ME includes three sequential but interdependent steps [5,6]: segmentation, symbol recognition and spatial relations interpretation. The aim of the segmentation process is to form the symbol hypotheses "h s " from the set of strokes.…”

Discovering real-world use cases for a multimodal math search interface

“…If users are visualizing a formally notated math expression before they search, it seems unfortunate to break this visual flow by forcing users to first convert the diagrammatic math expression into a coded, sentential one or build it through the tedious use of selection menus in order to initiate the search process. Further, since many math expressions are learned, consumed and recalled visually, being able to query using expressions (i.e., query-by-expression) seems a natural way for non-experts to search for mathematical information [12].…”

Rendering expressions to improve accuracy of relevance assessment for math search

“…Activity in the area of Mathematical Information Retrieval (MIR) has been increasing in recent years [15]. In 2013 a math retrieval competition was held as part of the NTCIR Workshop [1] and advances have been made in techniques for retrieving expressions by visual appearance [8,9] and operator structure [11].…”