MT3: Multi-Task Multitrack Music Transcription

Abstract

Automatic Music Transcription (AMT), inferring musical notes from raw audio, is a challenging task at the core of music understanding. Unlike Automatic Speech Recognition (ASR), which typically focuses on the words of a single speaker, AMT often requires transcribing multiple instruments simultaneously, all while preserving fine-scale pitch and timing information. Further, many AMT datasets are "low-resource", as even expert musicians find music transcription difficult and time-consuming. Thus, prior work has focused on task-specific architectures, tailored to the individual instruments of each task. In this work, motivated by the promising results of sequence-to-sequence transfer learning for low-resource Natural Language Processing (NLP), we demonstrate that a general-purpose Transformer model can perform multi-task AMT, jointly transcribing arbitrary combinations of musical instruments across several transcription datasets. We show this unified training framework achieves high-quality transcription results across a range of datasets, dramatically improving performance for low-resource instruments (such as guitar), while preserving strong performance for abundant instruments (such as piano). Finally, by expanding the scope of AMT, we expose the need for more consistent evaluation metrics and better dataset alignment, and provide a strong baseline for this new direction of multi-task AMT. 1

Jointist: Joint Learning for Multi-instrument Transcription and Its Applications

“…For f T , we propose a new instrument-wise metric to better capture the model performance for multi-instrument transcription. Existing literature uses mostly flat metrics or piece-wise evaluation [18,24,25,28]. Although this can provide a general idea of how good the transcription is, it does not show which musical instrument the model is particularly good or bad at.…”

“…Omnizart [16,29] is instrumentaware, but it does not scale up well when the number of musical instruments increases as discussed in Section 5.3. MT3 [18] is the current state-of-the-art MIAMT model. It formulates AMT as a sequence prediction task where the sequence consists of tokens of musical note representation.…”

“…In other words, to handle realistic use-cases of AMT, it is necessary to develop a multi-instrument transcription system. Recent examples are Omnizart [16,17] and MT3 [18] which we will discuss in Section 2.1.…”

A Comprehensive Review on Music Transcription

“…The grouped or separated stream typically corresponds to an individual instrument. Figure 1c shows an example of stream-level transcription which was obtained from a multi-task multitrack music transcription (MT3) model [31]. The estimated pitches and notes for each instrument in this model have been grouped into separate streams using various music transcription datasets.…”

“…The power of the transformer model is currently being used in many different fields of artificial intelligence including automatic music transcription. Inspired by the successful sequence-to-sequence transfer learning in natural language processing, one of the recent works demonstrates the effectiveness of a generalpurpose transformer model in transcribing various combinations of instruments across multiple datasets [31]. Additionally, another study takes the transformer model into account for the purpose of piano transcription.…”