SDR – Half-baked or Well Done?

Abstract

In speech enhancement and source separation, signal-to-noise ratio is a ubiquitous objective measure of denoising/separation quality. A decade ago, the BSS eval toolkit was developed to give researchers worldwide a way to evaluate the quality of their algorithms in a simple, fair, and hopefully insightful way: it attempted to account for channel variations, and to not only evaluate the total distortion in the estimated signal but also split it in terms of various factors such as remaining interference, newly added artifacts, and channel errors. In recent years, hundreds of papers have been relying on this toolkit to evaluate their proposed methods and compare them to previous works, often arguing that differences on the order of 0.1 dB proved the effectiveness of a method over others. We argue here that the signal-to-distortion ratio (SDR) implemented in the BSS eval toolkit has generally been improperly used and abused, especially in the case of single-channel separation, resulting in misleading results. We propose to use a slightly modified definition, resulting in a simpler, more robust measure, called scale-invariant SDR (SI-SDR). We present various examples of critical failure of the original SDR that SI-SDR overcomes.

Enhancing End-to-End Multi-Channel Speech Separation Via Spatial Feature Learning

“…Following the common speech separation metrics [12,21], we adopt average SI-SDR and SDR improvement over mixture as the evaluation metrics. We also report the performances under different ranges of angle difference between speakers to give a more comprehensive assessment for the model.…”

“…Finally, the decoder reconstructs the separated speech waveform from the masked mixture encode for each speaker. To optimize the network end-to-end, scale-invariant signal-to-distortion ratio (SI-SDR) [12] is utilized as the training objective:…”

Improving Noise Robust Automatic Speech Recognition with Single-Channel Time-Domain Enhancement Network

“…Time-domain loss (TDL): For time-domain networks, we employ the classic signal-to-noise ratio (SNR) [23] as time-domain loss,…”

“…where θ are the model parameters, SNR = −10 log 10 ( ||x|| 2 ||x−x|| 2 ) is the SNR between the clean speech and the enhanced speech, and || · || 2 is the L 2 norm. We decided to use the classic SNR loss [23] instead of the scale-invariant SNR (SiSNR) used in the original Tas-Net [17], because training the network with SiSNR let the network freely change the level of the enhanced signal. With the SNR loss, the scale of the signals is preserved avoiding any scaling requirement when passing the signal to ASR.…”

Speech Enhancement Using Self-Adaptation and Multi-Head Self-Attention

“…The three composite measures CSIG, CBAK, and COVL are the popular predictor of the mean opinion score (MOS) of the target signal distortion, background noise interference, and overall speech quality, respectively [30]. In addition, as the standard metric in speech enhancement, we also evaluated scale-invariant SDR (SI-SDR) [31]. Table 1 shows the experimental results.…”