Memory Based Online Learning of Deep Representations from Video Streams

Abstract

We present a novel online unsupervised method for face identity learning from video streams. The method exploits deep face descriptors together with a memory based learning mechanism that takes advantage of the temporal coherence of visual data. Specifically, we introduce a discriminative feature matching solution based on Reverse Nearest Neighbour and a feature forgetting strategy that detect redundant features and discard them appropriately while time progresses. It is shown that the proposed learning procedure is asymptotically stable and can be effectively used in relevant applications like multiple face identification and tracking from unconstrained video streams. Experimental results show that the proposed method achieves comparable results in the task of multiple face tracking and better performance in face identification with offline approaches exploiting future information. Code will be publicly available.

Deep learning in video multi-object tracking: A survey

“…Sharma et al [67] used instead a Recurrent Rolling Convolution (RRC) CNN [68] and a SubCNN [69] to detect vehicles in videos recorded on a moving camera in the context of autonomous driving (see section 3.2.4). Pernici et al [70] used the Tiny CNN detector [71] in their face tracking algorithm, obtaining a better performance when compared to the Deformable Parts Model detector (DPM) [25], that does not use deep learning techniques.…”

“…The authors in [100] used a fine-tuned GoogLeNet on the ILSVRC CLS-LOC [101] dataset for pedestrians recognition. In [70], the authors reused the visual features extracted by the CNN-based detector, and the association was performed using a Reverse Nearest Neighbor technique [102]. Sheng et al [103] employed the convolutional part of GoogLeNet to extract appearance features, using the cosine distance between them to compute an affinity score between pairs of detections, and merging that information with motion prediction in order to compute an overall affinity which serves as edge cost in a graph problem.…”

Towards Open World Object Detection

“…Their approach extends Nearest Class Mean classifier to operate in an open world setting by re-calibrating the class probabilities to balance open space risk. [46] studies open world face identity learning while [63] proposed to use an exemplar set of seen classes to match them against a new sample, and rejects it in case of a low match with all previously known classes. However, they don't test on image classification benchmarks and study product classification in e-commerce applications.…”

Real-time Visual Object Tracking with Natural Language Description

“…However, having redundant visual exemplars not only slows down the tracker, but also makes the tracker become biased and eventually drift away from the target. Therefore, we adopt the reverse nearest neighbor algorithm [22,32] and add Z t to Z if the reverse nearest neighbor set of Z t with Z is an empty set. The rationale is that we add Z t to Z only if the new exemplar "looks" different to its past, and therefore the memory captures the temporal appearance variations of the target.…”