Conditioned Source Separation for Musical Instrument Performances

Slizovskaia, Olga; Haro, Gloria

doi:10.1109/taslp.2021.3082331

Cited by 23 publications

(13 citation statements)

References 52 publications

(139 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, there has been increased interest in TSE applications to speech [17], [38]- [41], music [15], [16], [18], [19], [42]- [46], and universal sounds [2], [11]- [14], [47], [48]. Various types of auxiliary clues have been proposed to identify the target in a sound mixture, including enrollment audio samples [12], [18], [19], [38], [39], [47], class labels [2], [11], [45], video signals of the target source [15], [42], [48], [49], and recently even onomatopoeia [14].…”

Section: B Target Sound Extractionmentioning

confidence: 99%

SoundBeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learning

Delcroix¹,

Vázquez²,

Ochiai³

et al. 2022

Preprint

View full text Add to dashboard Cite

In many situations, we would like to hear desired sound events (SEs) while being able to ignore interference. Target sound extraction (TSE) aims at tackling this problem by estimating the sound of target SE classes in a mixture while suppressing all other sounds. We can achieve this with a neural network that extracts the target SEs by conditioning it on clues representing the target SE classes. Two types of clues have been proposed, i.e., target SE class labels and enrollment sound samples similar to the target sound. Systems based on SE class labels can directly optimize embedding vectors representing the SE classes, resulting in high extraction performance. However, extending these systems to the extraction of new SE classes not encountered during training is not easy. Enrollment-based approaches extract SEs by finding sounds in the mixtures that share similar characteristics to the enrollment. These approaches do not explicitly rely on SE class definitions and can thus handle new SE classes. In this paper, we introduce a TSE framework, SoundBeam, that combines the advantages of both approaches. We also perform an extensive evaluation of the different TSE schemes using synthesized and real mixtures, which shows the potential of SoundBeam.

show abstract

Section: B Target Sound Extractionmentioning

confidence: 99%

SoundBeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learning

Delcroix¹,

Vázquez²,

Ochiai³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…A lot of previous work exists for specific domains, such as separating speech from either other speech sources [12,13] or separating speech from other sounds [14,15]. Besides speech, there also has been much work towards extracting individual instrument tracks from music [16,17,2]. Another related line of research performs audio source separation with the help of with other modalities, such as vision [3,18] or accelerometers [19].…”

Section: Related Workmentioning

confidence: 99%

“…There are various ways in which this task could be approached. Given the recent success of deep learning for audio processing [1,2,3,4], we propose to train a neural network for this task. A key aspect of our proposal is not to tie the sound to be extracted to any predefined collection of sound categories (such as, for example, the ontology defined by AudioSet [5]).…”

Section: Introductionmentioning

confidence: 99%

One-Shot Conditional Audio Filtering of Arbitrary Sounds

Gfeller

Roblek

Tagliasacchi

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

We consider the problem of separating a particular sound source from a single-channel mixture, based on only a short sample of the target source (from the same recording). Using SoundFilter, a waveto-wave neural network architecture, we can train a model without using any sound class labels. Using a conditioning encoder model which is learned jointly with the source separation network, the trained model can be "configured" to filter arbitrary sound sources, even ones that it has not seen during training. Evaluated on the FSD50k dataset, our model obtains an SI-SDR improvement of 9.6 dB for mixtures of two sounds. When trained on Librispeech, our model achieves an SI-SDR improvement of 14.0 dB when separating one voice from a mixture of two speakers. Moreover, we show that the representation learned by the conditioning encoder clusters acoustically similar sounds together in the embedding space, even though it is trained without using any labels.

show abstract

“…In our system, there is only a single ACCDOA vector output at each time instant, and the class represented by this output is determined based on an input describing the type of sound event we want to locate. Such conditioning-based approaches are often used in audio source separation [8] to isolate the sound from a specific musical instrument [9,10], speaker [11,12], or sound event [13]. Compared to a system that outputs all classes simultaneously, conditioning-based approaches require multiple passes at inference-time (one per class) in order to detect all classes, similarly to the class-specific models, although they do not require training multiple separate models.…”

Section: Introductionmentioning

confidence: 99%

Locate This, Not That: Class-Conditioned Sound Event DOA Estimation

Slizovskaia¹,

Wichern²,

Wang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Existing systems for sound event localization and detection (SELD) typically operate by estimating a source location for all classes at every time instant. In this paper, we propose an alternative class-conditioned SELD model for situations where we may not be interested in localizing all classes all of the time. This class-conditioned SELD model takes as input the spatial and spectral features from the sound file, and also a one-hot vector indicating the class we are currently interested in localizing. We inject the conditioning information at several points in our model using feature-wise linear modulation (FiLM) layers. Through experiments on the DCASE 2020 Task 3 dataset, we show that the proposed class-conditioned SELD model performs better in terms of common SELD metrics than the baseline model that locates all classes simultaneously, and also outperforms specialist models that are trained to locate only a single class of interest. We also evaluate performance on the DCASE 2021 Task 3 dataset, which includes directional interference (sound events from classes we are not interested in localizing) and notice especially strong improvement from the class-conditioned model.

show abstract

Conditioned Source Separation for Musical Instrument Performances

Cited by 23 publications

References 52 publications

SoundBeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learning

SoundBeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learning

One-Shot Conditional Audio Filtering of Arbitrary Sounds

Locate This, Not That: Class-Conditioned Sound Event DOA Estimation

Contact Info

Product

Resources

About