The signal separation evaluation campaign (2007–2010): Achievements and remaining challenges

Abstract

We present the outcomes of three recent evaluation campaigns in the field of audio and biomedical source separation. These campaigns have witnessed a boom in the range of applications of source separation systems in the last few years, as shown by the increasing number of datasets from 1 to 9 and the increasing number of submissions from 15 to 34. We first discuss their impact on the definition of a reference evaluation methodology, together with shared datasets and software. We then present the key results obtained over almost all datasets. We conclude by proposing directions for future research and evaluation, based in particular on the ideas raised during the related panel discussion at the Ninth International Conference on Latent Variable Analysis and Signal Separation (LVA/ICA 2010).

A summary of the REVERB challenge: state-of-the-art and remaining challenges in reverberant speech processing research

“…Although existing benchmark tasks and challenges [8][9][10] mainly focus on the noise robustness issue and sometimes only in a single-channel scenario, a particular novelty of the REVERB challenge is that it is carefully designed to test robustness against reverberation, based on both single-channel and multichannel recordings made under moderately noisy environments. Another novel feature of the challenge is that its entire evaluation is based on real recordings and simulated data, part of which has similar characteristics to real recordings.…”

Blind Nonparametric Determined and Underdetermined Signal Extraction Algorithm for Dependent Source Mixtures

“…After convergence of the proposed algorithm, the average results of the performance criteria evaluated by SIR over 10 experiments are shown in Table 7. The definition of SIR for images can be found in [38]. …”

Real-time Soundprism

“…However, the single-channel SS problem is the extreme example of the under-determined cases, in which sensors are fewer than sources. Therefore, a high SS quality is not reachable by blind SS methods [26]. In that sense, it is necessary to exploit the knowledge of additional information to improve the separation [10] [27].…”