End-to-End Neural Optical Music Recognition of Monophonic Scores

Abstract

Optical Music Recognition is a field of research that investigates how to computationally decode music notation from images. Despite the efforts made so far, there are hardly any complete solutions to the problem. In this work, we study the use of neural networks that work in an end-to-end manner. This is achieved by using a neural model that combines the capabilities of convolutional neural networks, which work on the input image, and recurrent neural networks, which deal with the sequential nature of the problem. Thanks to the use of the the so-called Connectionist Temporal Classification loss function, these models can be directly trained from input images accompanied by their corresponding transcripts into music symbol sequences. We also present the Printed Images of Music Staves (PrIMuS) dataset, containing more than 80,000 monodic single-staff real scores in common western notation, that is used to train and evaluate the neural approach. In our experiments, it is demonstrated that this formulation can be carried out successfully. Additionally, we study several considerations about the codification of the output musical sequences, the convergence and scalability of the neural models, as well as the ability of this approach to locate symbols in the input score.

From Optical Music Recognition to Handwritten Music Recognition: A baseline

“…We use the Symbol Error Rate (SER) [17,18,19] metric. Similarly to Word Error Rate (WER) [28], commonly used in text recognition community, SER is computed as the Levenshtein distance: the sum of edit operations that are needed to convert the output of our method into the groundtruth in terms of symbol insertions (I), substitutions (S) and deletions (D).…”

“…For example, Van der Wel et al [17] use Convolutional Neural Networks (CNNs) and sequenceto-sequence (seq2seq) models for recognizing monophonic printed music scores. Calvo-Zaragoza et al [18,19] also use a CNN to extract features from printed music scores and feed a Recurrent Neural Network. To avoid the alignment between the music score and the ground-truth data, they use the Connectionist Temporal Classification (CTC) loss function commonly used in speech and text recognition.…”

“…Printed dataset: we use a subset of PrIMuS dataset [19], which consists of rendered incipts from the RISM 4 . It is annotated at primitive level i.e.…”

A diff procedure for music score files

“…The authors of [3], following the work of [17], observe that it does not make sense to apply directly the standard Unix diff utility to XML score files. A possible solution is to extract a linear representation of the graphical content [6], but motivating by the hierarchical structure of note beaming and tuplet grouping we chose to follow another approach and compare scores in terms of hierarchical structure, by using a tree-edit distance based on tree nodes operations, as proposed by [27] or [7].…”

“…At a detailed level, it is very valuable for musicologist and developers of version control systems to get precise clues on the locations of the differences between scores (e.g., between two editions of the same score). One difficulty that immediately arises for defining a diff tool for music scores is that, due to the nature/complexity of the music language, a music score contains multiple levels [6,10] that can be compared.…”