Facial Micro-expression Spotting and Recognition Using Time Contrasted Feature with Visual Memory

Nag, Sauradip; Bhunia, Ayan Kumar; Konwer, Aishik; Roy, Partha Pratim

doi:10.1109/icassp.2019.8683737

Cited by 11 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ELRCN [33], adopting a similar strategy to Kim et al, innovated by training CNN and LSTM jointly to ensure ME features' internal consistency and to decrease computational time. Nag et al [34] introduced a joint network to extract discriminative temporal features, distinguishing MEs from rapid muscle movements. Wang et al [35] introduced MESNet, which employs 2D convolution for spatial feature extraction and 1D convolution for modeling temporal relations.…”

Section: Deep Learning Methodsmentioning

confidence: 99%

FESNet: Spotting Facial Expressions Using Local Spatial Discrepancy and Multi-Scale Temporal Aggregation

Zhang,

Lu,

Wang

et al. 2024

cai

View full text Add to dashboard Cite

Facial expressions (FEs) spotting aims to split long videos into intervals of neutral expression, macro-expression, or micro-expression. Recent works mainly focus on feature descriptor or optical flow methods, suffering from difficulty capturing subtle facial motion and efficient temporal aggregation. This paper proposes a novel end-to-end network, named FESNet (Facial Expression Spotting Network), to solve the above challenges. The main idea is to model the subtle facial motion as local spatial discrepancy and incorporate temporal correlation by multi-scale temporal convolution. The FESNet comprises a local spatial discrepancy module (LSDM) and a multi-scale temporal aggregation module (MTAM). The LSDM first extracts the static spatial features from each frame by residual convolution and learns the inner spatial correlation by multi-head attention. Moreover, the subtle facial motion of facial expression is modeled as the discrepancy between the first frame and the current frame of the input interval, making frame-wise spatial proposals. Using the local spatial discrepancy features and proposals as input, the MTAM incorporates the temporal correlation by multi-scale temporal convolution and performs cascade refinement to make the final prediction. Furthermore, this paper proposes a smooth loss to ensure the temporal consistency of the cascade refined proposals from MTAM. Comprehensive experiments show that FESNet achieves competitive performance compared to state-of-the-art methods.

show abstract

Section: Deep Learning Methodsmentioning

confidence: 99%

FESNet: Spotting Facial Expressions Using Local Spatial Discrepancy and Multi-Scale Temporal Aggregation

Zhang,

Lu,

Wang

et al. 2024

cai

View full text Add to dashboard Cite

show abstract

“…Hong [14] used a sliding window to detect micro-expressions in samples with a fixed number of frames and treated micro-expression spotting as a binary classification task. Nag [15] proposed a joint architecture of temporal and spatial information to detect the onset frame and offset frame of microexpressions. Verburg M [16] applied the computed HOOF features into a recurrent neural network (RNN) for micro-expression localization, which combined deep learning and traditional methods and applied them to micro-expression spotting.…”

Section: Related Workmentioning

confidence: 99%

Micro-Expression Spotting Based on VoVNet, Driven by Multi-Scale Features

Yang,

Wu,

2023

Electronics

View full text Add to dashboard Cite

Micro-expressions are a type of real emotional expression, which are unconscious and difficult to hide. Identifying these expressions has great potential applications in areas such as civil aviation security, criminal interrogation, and clinical medicine. However, because of their characteristics such as short duration, low intensity, and sparse action units, this makes micro-expression spotting difficult. To address this problem and inspired by object detection methods, we propose a VoVNet-based micro-expression spotting model, driven by multi-scale features. Firstly, VoVNet is used to achieve the extraction and reuse of different scale perceptual field features to improve the feature extraction capability. Secondly, multi-scale features are extracted and fused using the Feature Pyramid Network module, incorporating optical flow features, and by realizing the interactive fusion of fine-grained feature information and semantic feature information. Finally, the model is trained and optimized on CAS(ME)2 and SAMM Long Video. The experimental results show that the F1 score of the proposed model is improved by 0.1963 and 0.2441 on the two datasets compared with the baseline method, which outperforms the most popular spotting methods.

show abstract

“…The DTSCNN was the first work in MER that utilized shallow two-stream neural network with inputs of optical-flow sequences. Nag et al [29] proposed an unified architecture for microexpression spotting and recognition, in which spatial and temporal network extracts time-contrasted features from the feature maps to contrast out subtle motions of micro-expressions. Wang et al proposed transferring Long-term Convolutional Neural Network (TL-CNN) [30] which utilized transfer learning from macroexpression to micro-expression database for MER.…”

Section: Features Evaluated On the Single Databasementioning

confidence: 99%

Feature refinement: An expression-specific feature learning and fusion method for micro-expression recognition

Zhou¹,

Mao²,

Huang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Micro-Expression Recognition has become challenging, as it is extremely difficult to extract the subtle facial changes of micro-expressions. Recently, several approaches proposed several expression-shared features algorithms for micro-expression recognition. However, they do not reveal the specific discriminative characteristics, which lead to sub-optimal performance. This paper proposes a novel Feature Refinement (FR) with expression-specific feature learning and fusion for micro-expression recognition. It aims to obtain salient and discriminative features for specific expressions and also predict expression by fusing the expression-specific features. FR consists of an expression proposal module with attention mechanism and a classification branch. First, an inception module is designed based on optical flow to obtain expression-shared features. Second, in order to extract salient and discriminative features for specific expression, expression-shared features are fed into an expression proposal module with attention factors and proposal loss. Last, in the classification branch, labels of categories are predicted by a fusion of the expressionspecific features. Experiments on three publicly available databases validate the effectiveness of FR under different protocol. Results on public benchmarks demonstrate that our FR provides salient and discriminative information for micro-expression recognition. The results also show our FR achieves better or competitive performance with the existing state-of-the-art methods on micro-expression recognition.

show abstract

Facial Micro-expression Spotting and Recognition Using Time Contrasted Feature with Visual Memory

Cited by 11 publications

References 16 publications

FESNet: Spotting Facial Expressions Using Local Spatial Discrepancy and Multi-Scale Temporal Aggregation

FESNet: Spotting Facial Expressions Using Local Spatial Discrepancy and Multi-Scale Temporal Aggregation

Micro-Expression Spotting Based on VoVNet, Driven by Multi-Scale Features

Feature refinement: An expression-specific feature learning and fusion method for micro-expression recognition

Contact Info

Product

Resources

About