Margin based knowledge distillation for mobile face recognition

Nekhaev, Dmitry; Milyaev, Sergey; Laptev, Ivan

doi:10.1117/12.2557244

Cited by 6 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Knowledge distillation [80], [81] is a useful method of transferring knowledge from large teacher models to the smaller student models. There are several knowledge distillation methods, suitable for face recognition models [82]- [84], but they are not adapted to the Prototype Memory approach. For example, to initialize the training, [83] needs class centers for each class, calculated by teacher network.…”

Section: F Prototype Memory Knowledge Distillationmentioning

confidence: 99%

Prototype Memory for Large-Scale Face Representation Learning

Smirnov¹,

Garaev²,

Galyuk³

2022

IEEE Access

View full text Add to dashboard Cite

Face representation learning using datasets with a massive number of identities requires appropriate training methods. Softmax-based approach, currently the state-of-the-art in face recognition, in its usual "full softmax" form is not suitable for datasets with millions of persons. Several methods, based on the "sampled softmax" approach, were proposed to remove this limitation. These methods, however, have a set of disadvantages. One of them is a problem of "prototype obsolescence": classifier weights (prototypes) of the rarely sampled classes, receive too scarce gradients and become outdated and detached from the current encoder state, resulting in incorrect training signals. This problem is especially serious in ultra-large-scale datasets. In this paper, we propose a novel face representation learning model called Prototype Memory, which alleviates this problem and allows training on a dataset of any size. Prototype Memory consists of the limited-size memory module for storing recent class prototypes and employs a set of algorithms to update it in appropriate way. New class prototypes are generated on the fly using exemplar embeddings in the current mini-batch. These prototypes are enqueued to the memory and used in a role of classifier weights for softmax classification-based training. To prevent obsolescence and keep the memory in close connection with the encoder, prototypes are regularly refreshed, and oldest ones are dequeued and disposed of. Prototype Memory is computationally efficient and independent of dataset size. It can be used with various loss functions, hard example mining algorithms and encoder architectures. We prove the effectiveness of the proposed model by extensive experiments on popular face recognition benchmarks.

show abstract

Section: F Prototype Memory Knowledge Distillationmentioning

confidence: 99%

Prototype Memory for Large-Scale Face Representation Learning

Smirnov¹,

Garaev²,

Galyuk³

2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Knowledge distillation is a useful method of transferring knowledge from large teacher models to the smaller student models. There are several knowledge distillation methods, suitable for face recognition models [78]- [80], but they are not adapted to the Prototype Memory approach. For example, to initialize the training, [79] needs class centers for each class, calculated by teacher network.…”

Section: F Prototype Memory Knowledge Distillationmentioning

confidence: 99%

Prototype Memory for Large-scale Face Representation Learning

Smirnov,

Garaev,

Galyuk

et al. 2021

Preprint

View full text Add to dashboard Cite

Face representation learning using datasets with massive number of identities requires appropriate training methods. Softmax-based approach, currently the state-of-the-art in face recognition, in its usual "full softmax" form is not suitable for datasets with millions of persons. Several methods, based on the "sampled softmax" approach, were proposed to remove this limitation. These methods, however, have a set of disadvantages. One of them is a problem of "prototype obsolescence": classifier weights (prototypes) of the rarely sampled classes, receive too scarce gradients and become outdated and detached from the current encoder state, resulting in an incorrect training signals. This problem is especially serious in ultra-large-scale datasets. In this paper, we propose a novel face representation learning model called Prototype Memory, which alleviates this problem and allows training on a dataset of any size. Prototype Memory consists of the limited-size memory module for storing recent class prototypes and employs a set of algorithms to update it in appropriate way. New class prototypes are generated on the fly using exemplar embeddings in the current mini-batch. These prototypes are enqueued to the memory and used in a role of classifier weights for usual softmax classification-based training. To prevent obsolescence and keep the memory in close connection with encoder, prototypes are regularly refreshed, and oldest ones are dequeued and disposed. Prototype Memory is computationally efficient and independent of dataset size. It can be used with various loss functions, hard example mining algorithms and encoder architectures. We prove the effectiveness of the proposed model by extensive experiments on popular face recognition benchmarks.

show abstract