Speech is Silver, Silence is Golden: What do ASVspoof-trained Models Really Learn?

Abstract

The ASVspoof Dataset is one of the most established datasets for training and benchmarking systems designed for the detection of spoofed audio and audio deepfakes. However, we observe an uneven distribution of silence length in dataset's training and test data, which hints at the target label: Bona-fide instances tend to have significantly longer leading + trailing silences than spoofed instances. This could be problematic, since a model may learn to only, or at least partially, base its decision on the length of the silence (similar to the issue with the Pascal VOC 2007 dataset, where all images of horses also contained a specific watermark [1]). In this paper, we explore this phenomenon in depth. We train a number of networks on only a) the length of the leading silence and b) with and without leading + trailing silence. Results show that models trained on only the length of the leading silence perform suspiciously well: They achieve up to 85% percent accuracy and an equal error rate (EER) of 0.15 on the 'eval' split of the data. Conversely, when training strong models on the full audio files, we observe that trimming silence during preprocessing dramatically worsens performance (EER increases from 0.03 to 0.15). This could indicate that previous work may, in part, have learned only to classify targets based on the length of silence. Consequently, it could mean that spoofing detection may not be as advanced as previous high-scores have led to believe. We hope that by sharing these results, the ASV community can further evaluate this phenomenon.

The Sound of Silence: Efficiency of First Digit Features in Synthetic Audio Detection

“…al. [13] shows that a bias can be found in the distribution of the lengths of leading and trailing silences in bonafide and synthetic speeches. Authors argue that most detectors are just probably discriminating between forged and bonafide samples by using this information.…”

“…Authors argue that most detectors are just probably discriminating between forged and bonafide samples by using this information. In order to bypass this problem, silent parts were removed from the signal, as suggested in [13] but this led to a big loss in performance.…”

“…From this filtering, only a few samples (less than 1%) were then removed since the number of silent values was not enough to obtain meaningful statistics. Arguably, this is not an issue since as shown in [13], the very low amount of silence in the audio track allows an easy detection of synthetic audio samples. Moreover, computing FD statistics on a limited amount of signal windows would lead to highly irregular statistics: this implies strong divergences/distances with respect to Benford's law (and therefore, a correct classification).…”

“…Although these solutions aim at detecting the peculiar characteristics of a fake audio signal, more recent works [13] have highlighted how synthetic speech algorithms prove to be effective in spoken parts but fail in generating realistic silence. The work by Muller et al shows that the length of trailing silenced parts 1 in synthetic speech samples from ASVSpoof dataset [14] prove to have different statistics with respect to bonafide samples.…”

Does Audio Deepfake Detection Generalize?

“…attacks disjoint from the attacks seen in training. However, the test audios share some specific characteristics [26] with the training data, which is why model generalization cannot be judged using the 'eval' split of ASVspoof 2019 alone. This motivates the use of our proposed 'in-the-wild' dataset, c.f.…”