Deep Disturbance-Disentangled Learning for Facial Expression Recognition

Abstract

To achieve effective facial expression recognition (FER), it is of great importance to address various disturbing factors, including pose, illumination, identity, and so on. However, a number of FER databases merely provide the labels of facial expression, identity, and pose, but lack the label information for other disturbing factors. As a result, many methods are only able to cope with one or two disturbing factors, ignoring the heavy entanglement between facial expression and multiple disturbing factors. In this paper, we propose a novel Deep Disturbance-disentangled Learning (DDL) method for FER. DDL is capable of simultaneously and explicitly disentangling multiple disturbing factors by taking advantage of multi-task learning and adversarial transfer learning. The training of DDL involves two stages. First, a Disturbance Feature Extraction Model (DFEM) is pre-trained to perform multi-task learning for classifying multiple disturbing factors on the large-scale face database (which has the label information for various disturbing factors). Second, a Disturbance-Disentangled Model (DDM), which contains a global shared sub-network and two task-specific (i.e., expression and disturbance) sub-networks, is learned to encode the disturbance-disentangled information for expression recognition. The expression sub-network adopts a multi-level attention mechanism to extract expression-specific features, while the disturbance sub-network leverages adversarial transfer learning to extract disturbance-specific features based on the pre-trained DFEM. Experimental results on both the in-the-lab FER databases (including CK+, MMI, and Oulu-CASIA) and the in-the-wild FER databases

A Robust Facial Expression Recognition Algorithm Based on Multi-Rate Feature Fusion Scheme

“…The accuracy comparisons of each method using the CK+ database is shown in Table 11. For the CK+ database, the proposed scheme which was denoted as the bold face, did not get the best result compared with some existing methods [26,[68][69][70].…”

“…3DIR [65] 93.21 STCNN-CRF [66] 93.04 CNN-CTSLSTM [67] 93.90 DDL [68] 99.16 DJSTN [26] 99.21 FDRL [69] 99.54 MC-DCN [70] 95.50 Proposed Scheme 96. 23 The results from the 10-times trial on the MMI dataset are in Table 12.…”

“…3DIR [65] 77.50 STCNN-CRF [66] 68.51 CNN-CTSLSTM [67] 78.4 DDL [68] 83.67 DJSTN [26] 87.88 FDRL [69] 85.23 MC-DCN [70] 78.6 Proposed Scheme 96.69…”

D³Net: Dual-Branch Disturbance Disentangling Network for Facial Expression Recognition

“…Recently, with the rapid development of deep learning, facial expression recognition (FER) has made remarkable progress in multimedia and computer vision [32,36,39,40,45], mainly due to its practical significance in human-computer interaction, health care systems, digital entertainment, etc [6]. However, FER is still a very challenging problem.…”

“…In Figure 1(a), variations of identity, pose, illumination, gender, race, and age exist in facial images and interfere the extraction of expression features. Extensive deep learning-based FER methods [4,30,32,43] have been proposed to exploit the labels of common disturbing factors and explicitly disentangle the disturbance.…”