Learning Audio–Sheet Music Correspondences for Cross-Modal Retrieval and Piece Identification

Abstract

This work addresses the problem of matching musical audio directly to sheet music, without any higherlevel abstract representation. We propose a method that learns joint embedding spaces for short excerpts of audio and their respective counterparts in sheet music images, using multimodal convolutional neural networks. Given the learned representations, we show how to utilize them for two sheet-music-related tasks: (1) piece/score identification from audio queries and (2) retrieving relevant performances given a score as a search query. All retrieval models are trained and evaluated on a new, large scale multimodal audio-sheet music dataset which is made publicly available along with this article. The dataset comprises 479 precisely annotated solo piano pieces by 53 composers, for a total of 1,129 pages of music and about 15 hours of aligned audio, which was synthesized from these scores. Going beyond this synthetic training data, we carry out first retrieval experiments using scans of real sheet music of high complexity (e.g., nearly the complete solo piano works by Frederic Chopin) and commercial recordings by famous concert pianists. Our results suggest that the proposed method, in combination with the large-scale dataset, yields retrieval models that successfully generalize to data way beyond the synthetic training data used for model building.

Learning Audio–Sheet Music Correspondences for Cross-Modal Retrieval and Piece Identification

“…The approach presented by Dorfer et al (2017a) and further exended in the present article goes beyond that of Dorfer et al (2016) in several respects. Most importantly, the original network required both sheet music and audio as input at the same time, in order to then decide which location in the sheet image best matches the current audio excerpt.…”

“…In the present work we continue the work of Dorfer et al (2017a) and extend it with the following new contributions, which we hope will greatly facilitate and accelerate future music alignment and retrieval research in the MIR community.…”

“…The central idea of this approach is to circumvent the problematic definition of mid-level features by replacing it (on both sides) with a learned transformation of audio and sheet music data to a common vector space. Dorfer et al (2017a) demonstrated how to utilize this methodology for two sheet music-related real-world applications: (1) piece identification via cross-modality retrieval from audio queries, and (2) audio-to-sheet music alignment using Dynamic Time Warping (DTW) in the learned joint embedding space.…”

“…Contribution 1: A New, Large, Open Multimodal Dataset. First experiments by Dorfer et al (2017a) already indicated that the general approach seems to scale very well with the amount of training data available, and that it is important to have as diverse a dataset as possible to arrive at a robust model. (To that end they also applied various data augmentation strategies.)…”

On the Adoption of Standard Encoding Formats to Ensure Interoperability of Music Digital Archives: The IEEE 1599 Format

“…Concerning the third typical case of music stakeholders, many datasets exist for MIR tasks, but they often lack the ability of interoperation. For instance, [16] contains source-separated and mixed audio and video tracks, MIDI scores and frame-level transcriptions; in [12] a dataset containing audio recordings, music scores and sheet music images is used; another interesting multimodal dataset containing time-aligned notes, audio and lyrics is presented in [18]; audio recordings, notes and expressive markings were recently collected in [15]. To date, each of these datasets used its own format for representing the synchronization of music along the various modalities.…”