Watch-List Screening Using Ensembles Based on Multiple Face Representations

Abstract

Still-to-video face recognition (FR) is an important function in watchlist screening, where faces captured over a network of video surveillance cameras are matched against reference stills of target individuals. Recognizing faces in a watchlist is a challenging problem in semi-and unconstrained surveillance environments due to the lack of control over capture and operational conditions, and to the limited number of reference stills. This paper provides a performance baseline and guidelines for ensemble-based systems using a single high-quality reference still per individual, as found in many watchlist screening applications. In particular, modular systems are considered, where an ensemble of template matchers based on multiple face representations is assigned to each individual of interest. During enrollment, multiple feature extraction (FE) techniques are applied to patches isolated in the reference still to generate diverse face-part representations that are robust to various nuisance factors (e.g., illumination and pose) encountered in video surveillance. The selection of relevant feature subsets, decision thresholds, and fusion functions of ensembles are achieved using faces of non-target individuals selected from reference videos (forming a universal background model). During operations, a face tracker gradually regroups faces captured from different people appearing in a scene, while each user-specific ensemble generates a decision per face capture. This leads to robust spatio-temporal FR when accumulated ensemble predictions surpass a detection threshold. Simulation results obtained with the Chokepoint video dataset show a significant improvement to accuracy, (1) when performing scorelevel fusion of matchers, where patches-based and FE techniques generate ensemble diversity; (2) when defining feature subsets and decision thresholds for each individual matcher of an ensemble using non-target videos; and (3) when accumulating positive detections over multiple frames.

Adaptive appearance model tracking for still-to-video face recognition

“…Among the SSPP techniques, multiple face representations are complex because they require multiple face descriptors and patch schemes within a common framework [27]. Though these techniques achieve some level of robustness against illumination variation or partial occlusion by using different features, they do not address pose or expression changes during operation.…”

“…The AAMT-FR technique proposed in this paper is compared to the systems for still-to-video FR based on template matching (TM-FR [8]), adaptive biometrics (TM-FR with self-update [9]), Sparse Variation Dictionary Learning (SVDL [41]), and Multiple Face Representation (MFR [27]). In TM-FR, input ROI patterns are extracted from the ROIs detected in a frame and compared with all the gallery-face-models using some similarity measure.…”

“…Each ROI may be extracted using different patches or sub-images to provide multiple representations of a face [27]. Several feature extraction techniques have been proposed in computer vision and in pattern recognition literature for FR, such as LBP, HOG, and PCA, used for FR [27,28].…”

Robust watch-list screening using dynamic ensembles of SVMs based on multiple face representations

“…The performance of the still-to-video FR systems designed according to the proposed framework are compared to reference systems [6], [41], [53] using videos from the publicly-available Chokepoint [50] and COX-S2V [22] datasets. Accuracy and efficiency are measured at the transaction-level (matching of input probe ROI against reference ROI) and at the trajectory-level (the entire FR system over multiple frames).…”

“…To design still-to-video FR systems from a limited number of reference face stills, a diversified pool of base classifiers can be generated to design an individual-specific ensemble through multiple representations. Multiple face representations of a single target ROI pattern has been shown to significantly improve the overall performance of basic template-based still-to-video FR system [6], [33]. Moreover, modular systems designed using individual-specific ensembles have been successfully applied to the detection of target individuals in VS [41], [47].…”