First Stereo Audio Source Separation Evaluation Campaign: Data, Algorithms and Results

Abstract

Abstract. This article provides an overview of the first stereo audio source separation evaluation campaign, organized by the authors. Fifteen underdetermined stereo source separation algorithms have been applied to various audio data, including instantaneous, convolutive and real mixtures of speech or music sources. The data and the algorithms are presented and the estimated source signals are compared to reference signals using several objective performance criteria.

The 2008 Signal Separation Evaluation Campaign: A Community-Based Approach to Large-Scale Evaluation

“…Four distinct datasets were provided: D1 Under-determined speech and music mixtures This dataset consists of 36 instantaneous, convolutive and recorded stereo mixtures of three to four audio sources of 10 s duration, sampled at 16 kHz. Recorded mixtures were acquired in a chamber with cushion walls, using the loudspeaker and microphone arrangement depicted in [1], while convolutive mixtures were obtained with artificial room impulse responses simulating the same arrangement. The distance between microphones was set to either 5 cm or 1 m and the room reverberation time (RT) to 130 ms or 250 ms.…”

“…Since the sources can be characterized only up to an arbitrary permutation, all possible permutations were tested and the one maximizing the average MER was selected. Tasks T3 and T4 were evaluated via the criteria in [6] and [1], respectively, termed signal to distortion ratio (SDR), source image to spatial distortion ratio (ISR), signal to interference ratio (SIR) and signal to artifacts ratio (SAR). These criteria can be computed for any separation system and do not necessitate knowledge of the unmixing filters or masks.…”

Nonparametric Bayesian sparse factor analysis for frequency domain blind source separation without permutation ambiguity

“…We also evaluated our method in terms of the signalto-distortion ratio (SDR), the image-to-spatial distortion ratio (ISR), the source-to-interference ratio (SIR), and the source-to-artifacts ratio (SAR) [26]. SDR is an overall measure of the separation performance; ISR is a measure of the correctness of the inter-channel information; SIR is a measure of the suppression of the interference signals; and SAR is a measure of the naturalness of the separated signals.…”

The signal separation evaluation campaign (2007–2010): Achievements and remaining challenges

“…Early criteria applied to biomedical data or toy audio data [13,14,16] were restricted to linear unmixing or binary time-frequency masking and required knowledge of the unmixing filters or the time-frequency masks. More recently, a family of criteria has been proposed that applies to all mixtures and algorithms [17,6]. In the case of source spatial image estimation, the criteria derive from the decomposition of an estimated source imageŝ img ij (t) as [6] …”

“…Regular evaluation has become necessary to reveal the effects of different algorithm designs, specify a common evaluation methodology and promote new results in other research communities and in the industry. It is with these objectives in mind that several evaluation campaigns have been held in the last few years, including the 2007 Stereo Audio Source Separation Evaluation Campaign (SASSEC) [6] and the 2008 and 2010 Signal Separation Evaluation Campaigns (SiSEC) [7,8,9] While SASSEC was restricted to audio and fully specified by the organizers, the two SiSEC campaigns were open to all application areas and organized in a collaborative fashion. A few initial datasets, tasks and evaluation criteria were proposed by the organizers.…”