Facial Expression Recognition via Joint Deep Learning of RGB-Depth Map Latent Representations

Abstract

Humans use facial expressions successfully for conveying their emotional states. However, replicating such success in the human-computer interaction domain is an active research problem. In this paper, we propose deep convolutional neural network (DCNN) for joint learning of robust facial expression features from fused RGB and depth map latent representations. We posit that learning jointly from both modalities result in a more robust classifier for facial expression recognition (FER) as opposed to learning from either of the modalities independently. Particularly, we construct a learning pipeline that allows us to learn several hierarchical levels of feature representations and then perform the fusion of RGB and depth map latent representations for joint learning of facial expressions. Our experimental results on the BU-3DFE dataset validate the proposed fusion approach, as a model learned from the joint modalities outperforms models learned from either of the modalities.

Learning to Fuse Latent Representations for Multimodal Data

“…BU-3DFE dataset contains both 3D and 2D data modalities for facial expressions of 100 different subjects. The data processing in [14] is followed for preparing the training data. DepthMap-ResNet50 [14] 61.11 DepthMap-VGG19 [14] 28.06 DepthMap-scratch [14] 84.72 RGB-ResNet50 [14] 82.92 RGB-VGG19 [14] 81.…”

“…DepthMap-ResNet50 [14] 61.11 DepthMap-VGG19 [14] 28.06 DepthMap-scratch [14] 84.72 RGB-ResNet50 [14] 82.92 RGB-VGG19 [14] 81. [16] 82.30 Distance+slopes+SVM [18] 87.10 2D+3D features fusion+SVM [19] 86.32 Geometric scattering representation+SVM [20] 84.80 Geometric+photometric attributes+VGG19 [21] 84.87 NF:RGB-ResNet50+DepthMap-scratch [14] 87.08 NF: RGB-VGG19+DepthMap-scratch [14] 89.31 Ours: RGB-ResNet50+DepthMap-scratch 89.86 Ours: RGB-VGG19+DepthMap-scratch 90.69 Table 2. Results comparison on BU-3DFE dataset Also, a similar experimental setting is used for extracting latent expresentations from both data modalities; that is, using pre-trained models (ResNet-50 and VGG19) on ImageNet dataset for RGB data and training a DNN from scratch on depth data.…”

“…Results comparison on BU-3DFE dataset Also, a similar experimental setting is used for extracting latent expresentations from both data modalities; that is, using pre-trained models (ResNet-50 and VGG19) on ImageNet dataset for RGB data and training a DNN from scratch on depth data. Furthermore, 10-fold cross-validation (CV) is employed for evaluating the models as in [14,15,16,17]. The results are given in Table 1, along with models trained on RGB only, depth map only and via naive fusion as in [14].…”

“…Furthermore, 10-fold cross-validation (CV) is employed for evaluating the models as in [14,15,16,17]. The results are given in Table 1, along with models trained on RGB only, depth map only and via naive fusion as in [14]. It will be seen that the proposed fusion approach gives improved results over naive fusion.…”

3DBodyTex: Textured 3D Body Dataset

“…The two pieces of information combined provide more dimensions to model and process, e.g. [9,26,16]. This richer information is desired in several scenarios.…”