An interactive audio source separation framework based on non-negative matrix factorization

Abstract

Though audio source separation offers a wide range of applications in audio enhancement and post-production, its performance has yet to reach the satisfactory especially for single-channel mixtures with limited training data. In this paper we present a novel interactive source separation framework that allows end-users to provide feedback at each separation step so as to gradually improve the result. For this purpose, a prototype graphical user interface (GUI) is developed to help users annotating time-frequency regions where a source can be labeled as either active, inactive, or well-separated within the displayed spectrogram. This user feedback information, which is partially new with respect to the state-of-the-art annotations, is then taken into account in a proposed uncertainty-based learning algorithm to constraint the source estimates in next separation step. The considered framework is based on non-negative matrix factorization and is shown to be effective even without using any isolated training data.Index Terms-Interactive audio source separation, nonnegative matrix factorization, uncertainty-based learning, timefrequency annotation, user feedback.

On-the-Fly Audio Source Separation—A Novel User-Friendly Framework

“…Such information can be e.g., user-"hummed" sounds that mimic the sources in the mixture [13] or source activity annotation along time [14] or in a time-frequency plane [15]; the annotation information is then used, instead of training data, to guide the separation process. Furthermore, recent publications disclose an interactive strategy [16], [17] where the user can perform annotations on the spectrogram of intermediate separation results to gradually correct the remaining errors. Note however that most of the existing approaches need to use prior information which may not be easy to acquire in advance (e.g., musical score, text transcript), is difficult to produce (e.g., user-hummed examples), or simply requires very experienced users while being very time consuming (e.g., time-frequency annotations).…”

Text-Informed Audio Source Separation. Example-Based Approach Using Non-Negative Matrix Partial Co-Factorization

“…In summary, this strategy consists in first updating locally all the entries of one matrix using the corresponding update among (12), (13) and (15), and in choosing one entry per column yielding the highest likelihood while setting to zero all the other entries (see [20] for more details). This strategy guarantees a local optimization of the cost (11) in the sense that the cost is guaranteed to remain non-increasing after each update.…”

“…It consists in using some auxiliary information about the sources and/or the mixing process to guide the separation. For example, score-informed approaches rely on musical score to guide the separation in music recordings [3][4][5][6], separation-by-humming (SbH) algorithms exploit a sound "hummed" by the user mimicking the source of interest [7,8], and user-guided approaches take into account knowledge about, e.g., user-selected F0 track [9] or userannotated source activity patterns along the spectrogram of the mixture [10,11] and/or that of the estimated sources [12,13]. In line with this direction, there are also speech separation systems informed, e.g., by speaker gender [14], by corresponding video [15], or by the natural language structure [16].…”

Multi-Channel Audio Source Separation Using Multiple Deformed References

“…It is well adapted to scenarios where the original sources are not available but high separation quality is nevertheless required. The additional information can be of different types: spatial and spectral information about the sources [5], [6], language structure [7], visual information [8], information about the recording/mixing conditions [9], musical scores [10]- [13], or user input [14]- [21]. For instance, the user can provide relevant information by drawing the fundamental frequency curve [18], by uttering the same sentence [16], by humming the melody [14], or even by selecting specific areas in the spectrogram of the mixture [17].…”