Short-Time Spectral Aggregation for Speaker Embedding

Abstract

State-of-the-art speaker verification systems take frame-level acoustics features as input and produce fixed-dimensional embeddings as utterance-level representations. Thus, how to aggregate information from frame-level features is vital for achieving high performance. This paper introduces short-time spectral pooling (STSP) for better aggregation of frame-level information. STSP transforms the temporal feature maps of a speaker embedding network into the spectral domain and extracts the lowest spectral components of the averaged spectrograms for aggregation. Benefiting from the low-pass characteristic of the averaged spectrograms, STSP is able to preserve most of the speaker information in the feature maps using a few spectral components only. We show that statistics pooling is a special case of STSP where only the DC spectral components are used. Experiments on VoxCeleb1 and VOiCES 2019 show that STSP outperforms statistics pooling and multi-head attentive pooling, which suggests that leveraging more spectral information in the CNN feature maps can produce highly discriminative speaker embeddings.

A Survey on Text-Dependent and Text-Independent Speaker Verification

“…Later, more advanced networks, such as ResNets [9], DenseNets [10], [12], and Res2Nets [11], [13], were introduced to better model the spectral-temporal relationship across the acoustic frames. Simultaneously, diverse aggregation methods have been proposed to aggregate the frame-level information into utterance-level embeddings, e.g., statistics pooling [2], multihead attentive pooling [14], NetVLAD-based pooling [9], short-time spectral pooling [15], [16], etc. Also, different training losses besides the softmax loss have been used in deep speaker embedding to achieve better discriminative power.…”

Aggregating Frame-Level Information in the Spectral Domain With Self-Attention for Speaker Embedding

“…However, because DFT can only be applied to deterministic or wide-sense stationary signals, it is not suitable for non-stationary speech signals [22]. To account for the nonstationarity of the convolutional feature maps in speaker embedding networks, short-time spectral pooling (STSP) was proposed in [23] by replacing DFT with short-time Fourier transform (STFT) [24]. It was shown in [23] that STSP is a generalized statistics pooling method.…”

“…To account for the nonstationarity of the convolutional feature maps in speaker embedding networks, short-time spectral pooling (STSP) was proposed in [23] by replacing DFT with short-time Fourier transform (STFT) [24]. It was shown in [23] that STSP is a generalized statistics pooling method. This is because from a Fourier perspective, statistics pooling only exploits the DC (zero-frequency) components in the spectral domain, whereas STSP incorporates more spectral components besides the DC ones during aggregation and is able to retain richer speaker information.…”

“…In attentive STSP, however, the pooling operation is performed completely in the spectral domain and inverse STFT is not required, which reduces computation and facilitates the aggregation. 2) Compared with vanilla STSP [23], attentive STSP applies a self-attention mechanism to highlight the contribution of the discriminative windowed segments in the pooling operation. The attention mechanism endows attentive STSP with higher discriminative power, and it is a new part not covered by [23].…”

“…2) Compared with vanilla STSP [23], attentive STSP applies a self-attention mechanism to highlight the contribution of the discriminative windowed segments in the pooling operation. The attention mechanism endows attentive STSP with higher discriminative power, and it is a new part not covered by [23].…”