Unsupervised Music Source Separation Using Differentiable Parametric Source Models

Abstract

Supervised deep learning approaches to underdetermined audio source separation achieve state-of-the-art performance but require a dataset of mixtures along with their corresponding isolated source signals. Such datasets can be extremely costly to obtain for musical mixtures. This raises a need for unsupervised methods. We propose a novel unsupervised modelbased deep learning approach to musical source separation. Each source is modelled with a differentiable parametric sourcefilter model. A neural network is trained to reconstruct the observed mixture as a sum of the sources by estimating the source models' parameters given their fundamental frequencies. At test time, soft masks are obtained from the synthesized source signals. The experimental evaluation on a vocal ensemble separation task shows that the proposed method outperforms learning-free methods based on nonnegative matrix factorization and a supervised deep learning baseline. Integrating domain knowledge in the form of source models into a data-driven method leads to high data efficiency: the proposed approach achieves good separation quality even when trained on less than three minutes of audio. This work makes powerful deep learning based separation usable in scenarios where training data with ground truth is expensive or nonexistent.

A Fully Differentiable Model for Unsupervised Singing Voice Separation

“…The original model proposed in [1], referred to as UMSS, was shown to be efficient for complex source separation problems in which individual sources are not available for training, the sources are homogeneous (only singing voices), or only a limited amount of mixture recording data is obtainable. The approach is inspired by the recent hybrid deep learning paradigm, which integrates signal processing models in DNNs to incorporate domain knowledge [9,10].…”

“…The latter strategy obtains the best overall results and is then selected in this work. The model used in [1] to obtain source fundamental frequencies was given in [11] and performs multi-F0 extraction by first processing a spectral representation through a DNN, and then converting the output multi-frequency salience map to F0 contours. In [1], a voice assignment heuristic based on temporal pitch continuity was ©2024 IEEE.…”

“…2. It is based on the unsupervised model described above [1] but with the integration of the multi-F0 estimation and vocal assignment as three differentiable blocks (displayed in purple on Fig. 2).…”

“…The resulting architecture is then fully differentiable and can be trained end-to-end. More precisely, the proposed architecture takes as input a 4second audio mixture which is processed in parallel in two branches: 1) the first branch is based on the encoder of [1], which extracts the main characteristics of the observed audio. 2) The second branch is dedicated to the estimation of multiple fundamental frequencies.…”

“…This paper builds upon the work of Schulze-Forster et al [1], which proposes an unsupervised DNN model that has shown state-of-the-art performance in separating choral singing. We expand their work by integrating the multi-F0 estimation and voice assignment modules as differentiable blocks within the model and by proposing a novel method for differentiable F0 contour extraction.…”