GEME: Dual-stream multi-task GEnder-based micro-expression recognition

Deep Learning for Micro-expression Recognition: A Survey

“…As the facial expression is easy to be influenced by the environment, pre-processing involving face detection and alignment is requiblack for robust facial expression recognition. Compablack with common facial expressions, MEs have low MER with DL Transfer learning: Finetune [54], [72], [94], [112] KD [67], [128], [129] Domain adaption [130]- [132] Multi-task: landmark [127] Gender [74] AU [120] Multi-binary-class [109] Loss: Cross-entroy [] Metric [121], [122] Margin [123]- [125] Imbalance [120], [126] Graph network: TCNGraph [118] ConvGraph [53], [119], [120] Network struc Cascade: CNN+RNN [111] CNN+LSTM [69], [105], [112]- [117] Multi-stream: Tradition+CNN: Dual [108]- [110] Different block: Dual [103]- [105] Triplet [106], [107] Same block: Dual [97]- [99] Triple [85], [100], [101] Four [102] Single stream: 2D [62], [72], [75] 3D [94]-...…”

“…The dynamic image has been successfully used in action recognition [152], as it allows to convert any video to an image so that the models pre-trained on large amount of still images can be immediately extended to videos. Currently, dynamic images have been utilized in micro-expressions [74]- [76] to summarize the subtle dynamics and appearance in an image for efficient micro-expression recognition. Furthermore, Liu et al [72] designed a dynamic segmented sparse imaging module (DSSI) to compute a set of dynamic images as the input data of subsequent models.…”

“…Among all these approaches, the focal loss [126] can be consideblack as the most simple and effective loss. MER task also suffers from data imbalance, the focal loss was utilized in MACNN [55], [74], [120], [179] to overcome the imbalance challenge. MER-auGCN [120] designed an adaptive factor with the focal loss to balance the proportion of the negative and positive samples in a given training batch.…”

“…Especially for MEs, the subtle facial movements are not able to contribute enough knowledge for robust MER. To address this issue, several deep learning [203] methods based multi-task learning have been subsequently developed for better MER [74], [127]. Multiple task learning [203] can learn better knowledge sharing among tasks, leading to better performance and low risk of overfitting in each task.…”

“…Following [127], several end-to-end deep multi-task networks leveraging different side tasks were proposed. GEME [74] contrusted a dual-stream multi-task network incorporating gender detection task while MER-auGCN [120] simultaneously detected AUs and MEs and further aggregated AU representation into ME representation through GCN. On the other hand, considering a common feature representation can be learned from multiple tasks, hu et al [109] formulated MER as a multi-task classification problem in which each category classification can be regarded as one-against-all pairwise classification problem.…”