Real-time face alignment: evaluation methods, training strategies and implementation optimization

Abstract

Face alignment is a crucial component in most face analysis systems. It focuses on identifying the location of several keypoints of the human faces in images or videos. Although several methods and models are available to developers in popular computer vision libraries, they still struggle with challenges such as insufficient illumination, extreme head poses, or occlusions, especially when they are constrained by the needs of real-time applications. Throughout this article, we propose a set of training strategies and implementations based on data augmentation, software optimization techniques that help in improving a large variety of models belonging to several real-time algorithms for face alignment. We propose an extended set of evaluation metrics that allow novel evaluations to mitigate the typical problems found in real-time tracking contexts. The experimental results show that the generated models using our proposed techniques are faster, smaller, more accurate, more robust in specific challenging conditions and smoother in tracking systems. In addition, the training strategy shows to be applicable across different types of devices and algorithms, making them versatile in both academic and industrial uses.

A High Performance and Robust FPGA Implementation of a Driver State Monitoring Application

“…Of course, this is an error extracted from the training set of photographs. If a similar error is extracted from a different test set, the average error is in the order of 5% which is comparable or smaller than the errors presented in [1] and [33]. The highest sensitivity is achieved by M1 (92.3%) and the highest precision by M2 (96.6%).…”

“…The speed achieved with each {ERT model, supporting hardware kernel} pair, measured in fps, is compared to some referenced approaches in Table 4 . In this table, the shortest frame processing latency achieved by the default model M0 on an Ubuntu i7 platform without the latency of the employed rules for higher robustness is also listed for comparison with the Local Binary Features (LBF) approach presented in [ 1 ]. In Table 4 , the resolution of the images and the number of landmarks aligned is also listed for a fair comparison.…”

“…Face landmark alignment is an operation exploited in several applications ([ 1 , 2 , 3 , 4 ]). More specifically, an interesting approach is presented in [ 3 ] where the authors use facial shape alignment to capture expressions that can predict strokes.…”

“…In the seminal work presented by Kowalski et al, in [ 33 ]. the proposed Deep Alignment Network (DAN) is actually a CNN for the alignment of 68 facial landmarks with a very small error (less than 5% according to [ 1 ]).…”

Automated gaze direction scoring from videos collected online through conventional webcam.

“…The individual frame images were processed in python, using the OpenCV (Bradski, 2000) and dlib (King, 2009) packages, and custom routines. Each individual frame image was first converted to greyscale then the bounding box of the face and individual eyes were extracted using dlib frontal face detector and 68-point face landmarks detector, respectively (Álvarez Casado & Bordallo López, 2021).…”