Attention-Based Temporal-Frequency Aggregation for Speaker Verification

Abstract

Convolutional neural networks (CNNs) have significantly promoted the development of speaker verification (SV) systems because of their powerful deep feature learning capability. In CNN-based SV systems, utterance-level aggregation is an important component, and it compresses the frame-level features generated by the CNN frontend into an utterance-level representation. However, most of the existing aggregation methods aggregate the extracted features across time and cannot capture the speaker-dependent information contained in the frequency domain. To handle this problem, this paper proposes a novel attention-based frequency aggregation method, which focuses on the key frequency bands that provide more information for utterance-level representation. Meanwhile, two more effective temporal-frequency aggregation methods are proposed in combination with the existing temporal aggregation methods. The two proposed methods can capture the speaker-dependent information contained in both the time domain and frequency domain of frame-level features, thus improving the discriminability of speaker embedding. Besides, a powerful CNN-based SV system is developed and evaluated on the TIMIT and Voxceleb datasets. The experimental results indicate that the CNN-based SV system using the temporal-frequency aggregation method achieves a superior equal error rate of 5.96% on Voxceleb compared with the state-of-the-art baseline models.

ResSKNet-SSDP: Effective and Light End-To-End Architecture for Speaker Recognition

“…Therefore, a flexible processing method should have the ability to accept audio of any duration and obtain the fixed-dimensional features. In speaker recognition systems, the GAP aggregation model [ 10 , 20 , 24 ] is the most commonly used method for aggregating the frame-level features into fixed-dimensional utterance-level features. The reference [ 7 ] employs statistical pooling (SP) to aggregate the features, which computes the mean vector of the frame-level features and the standard deviation vector of the second-order statistics; then, it stitches them together as an utterance-level feature.…”

“…It is calculated in the same way as the SP aggregation model, but has better performance than SP. They also introduced the NetVLAD aggregation model in computer vision that aggregates features into fixed dimensions via clustering [ 20 , 28 ]. NetVLAD assigns each frame-level feature to a different cluster center and encodes the residuals as output features.…”

“…Another challenge is the aggregation of frame-level features. In speaker recognition, the length of the input audio is variable, and an aggregation model is needed to aggregate the variable-length features into fixed-dimensional utterance-level features after the feature extractor has acquired the frame-level features [ 19 , 20 ]. The most common approach is to use global average pooling (GAP) directly to aggregate frame-level features into utterance-level features.…”

Application of Split Residual Multilevel Attention Network in Speaker Recognition

“…Then we can conclude that the receptive field is 1 after convolution with a 1×1 convolution kernel with a step length of 1. It is also possible to derive the per-layer receptive field of the Bottleneck Block, as shown in Equation (7), and the receptive field of parallel branches in the Split-ResNet Block, as shown in Equation (8). From the comparison results of Equation (7) and Equation ( 8), it can be concluded that Split-ResNet has more and larger receptive field, which makes Split-ResNet have stronger feature extraction ability than ResNet and can produce better results than the original network.…”

“…But with the development of deep learning, deep learning networks (DNNs) have brought breakthroughs in speaker recognition. The DNN architecture system is capable of directly processing the input audio data, then extracting the frame-level features of the input audio through DNN and aggregating the features through an aggregation model to aggregate the extracted frame-level features into discourse-level features, thus making deep learning dominant in the direction of speaker recognition [7], [8].…”