Face Flow

Abstract

In this paper, we propose a method for the robust and efficient computation of multi-frame optical flow in an expressive sequence of facial images. We formulate a novel energy minimisation problem for establishing dense correspondences between a neutral template and every frame of a sequence. We exploit the highly correlated nature of human expressions by representing dense facial motion using a deformation basis. Furthermore, we exploit the even higher correlation between deformations in a given input sequence by imposing a low-rank prior on the coefficients of the deformation basis, yielding temporally consistent optical flow. Our proposed model-based formulation, in conjunction with the inverse compositional strategy and lowrank matrix optimisation that we adopt, leads to a highly efficient algorithm for calculating facial flow. As experimental evaluation, we show quantitative experiments on a challenging novel benchmark of face sequences, with dense ground truth optical flow provided by motion capture data. We also provide qualitative results on a real sequence displaying fast motion and occlusions. Extensive quantitative and qualitative comparisons demonstrate that the proposed method outperforms state-of-the-art optical flow and dense non-rigid registration techniques, whilst running an order of magnitude faster.

A Comprehensive Performance Evaluation of Deformable Face Tracking “In-the-Wild”

“…Rigid and deformable tracking of faces and facial features have been a very popular topic of research over the past twenty years (Black and Yacoob 1995;Lanitis et al 1995;Sobottka and Pitas 1996;Essa et al 1996Essa et al , 1997Oliver et al 1997;Decarlo and Metaxas 2000;Jepson et al 2003;Xiao et al 2004;Patras and Pantic 2004;Kim et al 2008;Ross et al 2008;Papandreou and Maragos 2008;Amberg et al 2009;Kalal et al 2010a;Koelstra et al 2010;Tresadern et al 2012;Tzimiropoulos and Pantic 2013;Xiong and De la Torre 2013;Liwicki et al 2013;Smeulders et al 2014;Asthana et al 2014;Li et al 2016a;Xiong and De la Torre 2015;Snape et al 2015;Tzimiropoulos 2015). In this section we provide an overview of face tracking spanning over the past twenty years up to the present day.…”

“…Deformable tracking of faces can be further subdivided into i) tracking of certain facial landmarks (Lanitis et al 1995;Black and Yacoob 1995;Sobottka and Pitas 1996;Xiao et al 2004;Patras and Pantic 2004;Papandreou and Maragos 2008;Amberg et al 2009;Tresadern et al 2012;Xiong and De la Torre 2013;Asthana et al 2014;Xiong and De la Torre 2015) or ii) tracking/estimation of dense facial motion Yacoob and Davis 1996;Essa et al 1997;Decarlo and Metaxas 2000;Koelstra et al 2010;Snape et al 2015). The latter category of estimating a dense facial motion through a model-based system was proposed by MIT Media lab in mid 1990's (Essa et al 1997Basu et al 1996).…”

“…The performance of deformable dense facial tracking methodologies was usually assessed by using markers (Decarlo and Metaxas 2000), simulated data , visual inspection (Decarlo and Metaxas 2000;Essa et al 1997Essa et al , 1996Yacoob and Davis 1996;Snape et al 2015;Koelstra et al 2010) or indirectly by the use of the dense facial motion for certain tasks, such as expression analysis Yacoob and Davis 1996;Koelstra et al 2010). Regarding tracking of facial landmarks, up until recently, the preferred method for assessing the performance was visual inspection in a number of selected facial videos (Xiong and De la Torre 2013;Tresadern et al 2012).…”

“…Free-form deformations (Rueckert et al 1999) are used in Koelstra et al (2010) for extraction of dense facial motion for facial action unit recognition. Recently, Snape et al (2015) proposed a statistical model of the facial flow for fast and robust dense facial motion extraction.…”

Estimating Correspondences of Deformable Objects “In-the-Wild”

“…Furthermore, we capitalise on recent advances on multiframe optical flow estimation [31,49,55] and show that the relevant methodologies have matured enough to densely annotate the proposed shapes using either simplistic or even more sophisticated and robust shape representation methods [44]. In particular, in order to build dense correspondences between different shape instances of the same object class, we jointly estimate the optical flow among all the instances by imposing low-rank constrains, an approach that we call Shape Flow.…”