A joint hierarchical cross‐attention graph convolutional network for multi‐modal facial expression recognition

Abstract

Emotional recognition in conversations (ERC) is increasingly being applied in various IoT devices. Deep learning‐based multimodal ERC has achieved great success by leveraging diverse and complementary modalities. Although most existing methods try to adopt attention mechanisms to fuse different information, these methods ignore the complementarity between modalities. To this end, the joint cross‐attention model is introduced to alleviate this issue. However, multi‐scale feature information on different modalities is not utilized. Moreover, the context relationship plays an important role in feature extraction in the expression recognition task. In this paper, we propose a novel joint hierarchical graph convolution network (JHGCN) which exploits different layer features and context relationships for facial expression recognition based on audio‐visual (A‐V) information. Specifically, we adopt different deep networks to extract features from different modalities individually. For V modality, we construct V graph data based on patch embeddings which are extracted from the transformer encoder. Moreover, we embed the graph convolution which can leverage the intra‐modality relationships with the transformer encoder. Then, the deep feature from different layers is fed to the hierarchical fusion module to enhance feature representation. At last, we use the joint cross‐attention mechanism to exploit the complementary inter‐modality relationships. To validate the proposed model, we have conducted various experiments on the AffWild2 and CMU‐MOSI datasets. All results confirm that our proposed model achieves highly promising performance compared to the joint cross‐attention model and other methods.

Context Transformer and Adaptive Method with Visual Transformer for Robust Facial Expression Recognition

“…Considering the spatial localization of convolutional networks, researchers have proposed some methods to enhance the facial expression understanding of convolutional networks, such as the introduction of pyramid structure [25] to reduce the loss of effective information due to the deeper model or multi-scale convolutional mechanisms [26] to improve the ability to capture subtle changes in expressions. Some other researchers have used graph convolutional networks [10,11] to capture information about the expression features of dynamic sequences or to incorporate different attentional mechanisms depending on the task requirements, etc. Minaee et al [7] proposed an attention-based convolutional network focusing on different parts of the facial image to perform the FER task.…”

“…However, the spatial localization of convolutional networks makes it difficult for the model to learn the dependencies between different facial regions, thus limiting the understanding of global facial expressions. To overcome these problems, some improved models based on convolutional networks have also been proposed, such as the attention mechanism [7], graph convolutional networks [10,11], etc., which aim to enhance the model's ability to learn the dependencies between different facial regions, to better capture the global facial expression information and to improve the model's ability to recognize expressions. In recent years, the Transformer architecture [12] has achieved remarkable success in natural language processing tasks, and inspired by its successful application, ViT (Vision Transformer) [13] has been introduced into image classification tasks and achieved remarkable results through the non-local attention mechanism.…”