GhostVLAD for Set-Based Face Recognition

Abstract

The objective of this paper is to learn a compact representation of image sets for template-based face recognition. We make the following contributions: first, we propose a network architecture which aggregates and embeds the face descriptors produced by deep convolutional neural networks into a compact fixed-length representation. This compact representation requires minimal memory storage and enables efficient similarity computation. Second, we propose a novel GhostVLAD layer that includes ghost clusters, that do not contribute to the aggregation. We show that a quality weighting on the input faces emerges automatically such that informative images contribute more than those with low quality, and that the ghost clusters enhance the network's ability to deal with poor quality images. Third, we explore how input feature dimension, number of clusters and different training techniques affect the recognition performance. Given this analysis, we train a network that far exceeds the state-of-the-art on the IJB-B face recognition dataset. This is currently one of the most challenging public benchmarks, and we surpass the state-of-the-art on both the identification and verification protocols.

Utterance-level Aggregation for Speaker Recognition in the Wild

“…Therefore, while aggregating the frame-level features, the contribution of the noisy and undesirable sections of a speech segment to normal VLAD clusters is effectively downweighted, as most of their weights have been assigned to the 'ghost cluster'. For further details, please see [17].…”

“…We make the following contributions: (i) We propose a powerful speaker recognition deep network, based on a NetVLAD [16] or GhostVLAD [17] layer that is used to aggregate 'thin-ResNet' architecture frame features; (ii) The entire network is trained end-to-end using a large margin softmax loss on the large-scale VoxCeleb2 [3] dataset, and achieves a significant improvement over the current stateof-the-art verification performance on VoxCeleb1, despite using fewer parameters than the current state-of-the-art architectures [3,13]; and, (iii) We analyse the effect of input segment length on performance, and conclude that for 'in the wild' sequences having longer utterances (4s or more) is a significant improvement over shorter segments.…”

What Does a Network Layer Hear? Analyzing Hidden Representations of End-to-End ASR Through Speech Synthesis

“…The first model, following [17], which is a LSTM-based neural architecture, maps a sequence of variable-length mel-spectrogram frames to a fixed-dimensional embedding and is trained using the generalized end-to-end(GE2E) loss function. The second model, ThinResNet [16] is composed of a modified ResNet architecture, which extracts the features from the spectrogram of a speech utterance, and a NetVLAD/GhostVLAD [18] layer to aggregate the features along the temporal axis to an embedding. Both models are trained on the VoxCeleb2 [19] dataset, which is comprised of 1 million utterances from 6000 different speakers, and can achieve 4.51%, 3.34% EER respectively on the test-clean split of LibriSpeech [12] dataset.…”

Privacy Preserving Group Membership Verification and Identification

“…2.2.1). In a common approach, multiple face captures of the same person are combined to gain robustness against poses, expression, and quality variations [21]. In our scenario, the group is composed of unique faces of different persons.…”