Lighting and Pose Robust Face Sketch Synthesis

Abstract

Abstract. Automatic face sketch synthesis has important applications in law enforcement and digital entertainment. Although great progress has been made in recent years, previous methods only work under well controlled conditions and often fail when there are variations of lighting and pose. In this paper, we propose a robust algorithm for synthesizing a face sketch from a face photo taken under a different lighting condition and in a different pose than the training set. It synthesizes local sketch patches using a multiscale Markov Random Field (MRF) model. The robustness to lighting and pose variations is achieved in three steps. Firstly, shape priors specific to facial components are introduced to reduce artifacts and distortions caused by variations of lighting and pose. Secondly, new patch descriptors and metrics which are more robust to lighting variations are used to find candidates of sketch patches given a photo patch. Lastly, a smoothing term measuring both intensity compatibility and gradient compatibility is used to match neighboring sketch patches on the MRF network more effectively. The proposed approach significantly improves the performance of the state-of-the-art method. Its effectiveness is shown through experiments on the CUHK face sketch database and celebrity photos collected from the web.

Ensemble of Sparse Cross-Modal Metrics for Heterogeneous Face Recognition

“…The first category is the synthesis based methods which map data of one modality into another by synthetic methods. Related work includes synthesizing sketches from photos and then comparing synthesized images with sketches drawn by artists [31,36]. One drawback of this kind of methods is that different synthetic methods have to be used if the modalities change.…”

Data insufficiency in sketch versus photo face recognition

“…It can be done by either synthesizing sketches from photos or vice-versa. Many works have been proposed in this category including [4,12,13,[24][25][26]30,31,35]. Some of these methods report very good accuracy for sketchphoto recognition on CUFS dataset with appropriate parameters tuned.…”

“…Sketch images in the CUFSF are also hand-drawn in the same way as the CUFS dataset so there are now 1,800 sketch-photo pairs in the combined set of CUFS and CUFSF. There are many approaches that have been evaluated on the CUFS dataset including [4,8,9,13,21,[24][25][26][30][31][32]. Recent works have shown close to 100% recognition rate on the CUFS dataset [9,11,32] which implies that the problem is almost solved.…”

Face Sketch Synthesis System Utilizing Both Global Structure and Local Detailed Texture Components

“…These training samples are then used to learn how to generate (or exaggerate) a correct sketch for an unseen input photo. A typical kernel learning method is non-parametric-based Markov random field (MRF) [9,11,12]. For example, work [1] used the E-HMM approach to construct the relation between photo and sketch, where a human face is divided into five parts, namely, forehead, eye, nose, mouth, and chin, in order to preserve detailed textures.…”

“…Work [11] used the MRF graph to establish the correlation between the photo and sketch and constrains the smoothness between the neighborhood patches of the synthesized sketch. Work [12] was also based on the MRF approach, but this framework solved the varied light and pose changing conditions by using difference-of-Gaussian (DoG) representations. Work [9] integrated the parametric-based approach in the MRF structure.…”