HAMEX - A Handwritten and Audio Dataset of Mathematical Expressions

Abstract

In this paper, we present HAMEX, a new public dataset that contains mathematical expressions available in their on-line handwritten form and in their audio spoken form. We have designed this dataset so that, given a mathematical expression, its handwritten signal and its audio signal can be used jointly to design multimodal recognition systems. Here, we describe the different steps that allowed us to acquire this dataset, from the creation of the mathematical expression corpora (including expressions from Wikipedia pages) to the segmentation and the transcription of the collected data, via the data collection process itself. Currently, the dataset contains 4 350 on-line handwritten mathematical expressions written by 58 writers, and the corresponding audio expressions (in French) spoken by 58 speakers. The ground truth is also provided both for the handwritten expressions (as INKML files with the digital ink, the symbol segmentation, and the MATHML structure) and for the audio expressions (as XML files with the transcriptions of the spoken expressions).

Audio-video based character recognition for handwritten mathematical content in classroom videos

“…Some of the combination techniques proposed in our research are adaptations of well known methods such as weighted combination, Borda count [54], and other decision combination techniques [18]. In the context of handwriting and speech recognition, classifier combination techniques have also been used to improve the recognition accuracy of handwriting recognizers [14,41,48,59] as well as speech recognizers [12].…”

“…However, issues such as ambiguity detection and A/V synchronization are not considered in the aforementioned research. Another relevant research effort, closely tied to [33], relates to the creation of a data set with handwritten and spoken mathematical content [41]. Unfortunately, this data set consists of static image segments containing handwritten content, with the corresponding audio stored in separate files.…”

The Problem of Handwritten Mathematical Expression Recognition Evaluation

“…Training and test data for CROHME 2012 along with related tools were available from the International Association of Pattern Recognition (IAPR) 4 . For Part 4, the training data includes thousands of expressions from existing handwritten expression datasets, including (i) MathBrush (University of Waterloo) [18], (ii) HAMEX (University of Nantes) [19], (iii) MfrDB (Czech Technical University) [5], (iv) ExpressMatch (University of Sao Paulo) [20] and (v) the KAIST dataset. Due to differences in legal symbols and layouts, not all expressions in these data sets were consistent with the Part 4 grammar.…”

Database of Handwritten Arabic Mathematical Formula Images

“…The presence of several accessible corpora for the recognition enable this domain and it is useful for many fields, such as the field of Latin mathematical formula recognition. This field presents a datasets that facilitates the progress of this domain like the HAMEX [4]. The HAMEX is a public dataset that contains mathematical expressions in their handwritten form and in their audio spoken form.…”