Imponderous Net for Facial Expression Recognition in the Wild

Abstract

Since the renaissance of deep learning (DL), facial expression recognition (FER) has received a lot of interest, with continual improvement in the performance. Hand-in-hand with performance, new challenges have come up. Modern FER systems deal with face images captured under uncontrolled conditions (also called in-the-wild scenario) including occlusions and pose variations. They successfully handle such conditions using deep networks that come with various components like transfer learning, attention mechanism and local-global context extractor. However, these deep networks are highly complex with large number of parameters, making them unfit to be deployed in real scenarios. Is it possible to build a light-weight network that can still show significantly good performance on FER under in-the-wild scenario? In this work, we methodically build such a network and call it as Imponderous Net. We leverage on the aforementioned components of deep networks for FER, and analyse, carefully choose and fit them to arrive at Imponderous Net. Our Imponderous Net is a low calorie net with only 1.45M parameters, which is almost 50x less than that of a state-of-the-art (SOTA) architecture. Further, during inference, it can process at the real time rate of 40 frames per second (fps) in an intel-i7 cpu. Though it is low calorie, it is still power packed in its performance, overpowering other light-weight architectures and even few high capacity architectures. Specifically, Imponderous Net reports 87.09%, 88.17% and 62.06% accuracies on in-the-wild datasets RAFDB, FERPlus and AffectNet respectively. It also exhibits superior robustness under occlusions and pose variations in comparison to other light-weight architectures from the literature.

PAtt-Lite: Lightweight Patch and Attention MobileNet for Challenging Facial Expression Recognition

“…From the comparison with the best results on these subsets, the proposed PAtt-Lite achieved state-of-the-art performance across all subsets with significantly lesser parameters. Meanwhile, when compared with the lighter methods such as RAN [5] and Imponderous Net [10] on subsets of RAF-DB, PAtt-Lite managed to outperform these methods by around 9%. The proposed method also managed to outperform the lighter methods by more than 9% on the first two subsets and by 7.75% on the Pose 45 subset of FERPlus.…”

“…This corresponds to an improvement in the average accuracy of 4.02% over APViT [26], which previously reported the best average accuracy. Moreover, when compared to Imponderous Net [10], which has the nearest number of parameters as our proposed method, PAtt-Lite achieved an improvement of 12.81% in terms of average accuracy, while performing stronger and more consistent on all expression classes.…”

“…Furthermore, following [25]- [28], a per-class performance comparison is added in Table 9 to compare the performance of PAtt-Lite on different classes of the database. For [9], [10], [24], [29], [44], which did not specifically report their perclass accuracy, their results from the confusion matrices are taken for comparison. Overall, a similar trend for the perclass performance is observed when PAtt-Lite is compared with existing work, where the per-class accuracy for the Disgust class and the Fear class is lacking behind the other classes.…”

“…The patch extraction block is designed to solely extract local features. This distinguishes it from the methods used in [6], [8], [10], which employed patch attention mechanisms. Specifically, the proposed patch extraction is inspired by that of the vision transformer architecture but remains different from its inspiration in terms of implementation details, which will be explained in the next subsection.…”

A survey of facial expression recognition in the wild

“…The second part is the further extraction of features. Based on [16], two identical ParaFeat modules are set up for parallel operation. The final step is to use two fully connected global average pools (gaps) for feature classification.…”