Video Face Clustering With Unknown Number of Clusters

Abstract

Understanding videos such as TV series and movies requires analyzing who the characters are and what they are doing. We address the challenging problem of clustering face tracks based on their identity. Different from previous work in this area, we choose to operate in a realistic and difficult setting where: (i) the number of characters is not known a priori; and (ii) face tracks belonging to minor or background characters are not discarded.To this end, we propose Ball Cluster Learning (BCL), a supervised approach to carve the embedding space into balls of equal size, one for each cluster. The learned ball radius is easily translated to a stopping criterion for iterative merging algorithms. This gives BCL the ability to estimate the number of clusters as well as their assignment, achieving promising results on commonly used datasets. We also present a thorough discussion of how existing metric learning literature can be adapted for this task. 1 We consider three types of characters based on their roles: primary or recurring characters have major roles in several episodes; secondary or minor characters are named and play an important role in some episodes; and background or unknown (Unk) characters are unnamed and uncredited.

AVA-AVD: Audio-Visual Speaker Diarization in the Wild

“…Thus, they can generalize well on the unseen videos. Based on the pretrained models, many interesting downstream tasks have been explored, such as deep clustering [30,33], face clustering [34,61], person search [67], person clustering [4], as well as speaker diarization [10,41]. The above verification models are all uni-modal.…”

“…Backbone. Pretrained on large-scale datasets, speaker verification, and face recognition models have strong generalization ability and are directly applied in various downstream tasks [4,10,30,33,34,41,61,67]. We follow these works and utilize the backbones of off-the-shelf models [8,17] to encode voice and face features respectively.…”

“…Input features. Recent face clustering [61], multi-modal clustering [30,33], and verification [49] methods learn the threshold or unified representation based on the embedding vectors extracted by pretrained models. Hence, we also study the influence of inputs for our AVR-Net.…”

“…During inference, we use a common sliding window approach to process the input sequence segment by segment for videos of arbitrary length. We randomly select one image from the face tracks because, unlike averaging embedding vectors [61], averaging feature maps is harmful to our AVR-Net and thus does not bring any performance gain in practice. Moreover, cannot-link and must-link constraints are widely used by face clustering algorithms [4,34,61].…”

“…We randomly select one image from the face tracks because, unlike averaging embedding vectors [61], averaging feature maps is harmful to our AVR-Net and thus does not bring any performance gain in practice. Moreover, cannot-link and must-link constraints are widely used by face clustering algorithms [4,34,61]. However, this technique does not work in our system and is ignored consequently.…”