Warp that smile on your face: Optimal and smooth deformations for face recognition

Abstract

Abstract-In this work, we present novel warping algorithms for full 2D pixel-grid deformations for face recognition.Due to high variation in face appearance, face recognition is considered a very difficult task, especially if only a single reference image, for example a mug-shot, per face is available.Usually model-based approaches with additional training data are used to cope with several types of variation occurring in facial imaging. Image warping contrarily yields a distance measure which is invariant with regard to several types of variation. This allows for precise recognition even using only very few reference observations. Due to the computationally complex problem of optimal 2D warping, pseudo-2D warpingbased approaches in the past represented strong approximations of the original problem, and were mainly successful on data with low variability or rectified images.We propose a novel 2D warping method which is globally optimal and makes no prior assumtions on the data variability besides two-dimensional smootheness constraints which both avoid local mirroring and gaps and significantly speed up the optimization. Furthermore, we show that occlusion handling is imperative to obtain smooth warpings in a variety of domains.We evaluate our novel algorithm on various well known databases, such as the AR-Face and CMU-PIE database, and provide a detailed comparison to existing warping approaches. We show that by using simple relative 2D constraints, strong local features and a kernel, which is robust w.r.t. occlusions, our computationally complex approaches outperform state-ofthe-art results for recognizing faces under varying expressions, occlusions and poses. Most interestingly, we achieve higher accuracy using fewer training instances per class compared to methods learning a model of the 3D shape.

Pose-Invariant Face Recognition with a Two-Level Dynamic Programming Algorithm

“…We start by briefly reviewing the problem formulation of 2D warping in correspondence with [5] and [6]. Given a source image X = {x ij } with resolution I ×J and a target image R = {r uv } with resolution U × V , we search for a mapping (i, j) → w ij = (u, v) that assigns each pixel of the source image to a pixel of the target image with minimal energy cost.…”

“…We found that this distinction is not necessary and simply matching both half-images and selecting the lower resulting energy is sufficient for all poses. As in [12] and [6] we use the gallery image as source and the probe image as target.…”

“…Local context can be included by computing the cost-function over a small patch centered around the pixel (i, j) and the distances are pre-computed and cached [6]. To prevent outliers from having too much influence, the distance function is truncated by a threshold τ [6].…”

“…By this procedure, deformations caused by the previously mentioned variations can be taken into account. It has been shown that structural constraints on the image matching lead to smoother deformations and an increased recognition accuracy [5,6]. In order to incorporate such constraints and to apply relative penalty functions, the dependencies defined by a 2D grid need to be maintained.…”

“…For each column one tree is constructed, where vertical dependencies represent the stem and the horizontal dependencies represent the branches. In [6] this approach is extended by using structural constraints [5] (CTSDP). The before mentioned approaches suffer from the drawback that the final solution can only maintain vertical dependencies, due to the independent optimization of the columns.…”