Dual-stream Shallow Networks for Facial Micro-expression Recognition

Khor, Huai-Qian; See, John; Liong, Sze‐Teng; Phan, Raphaël C.‐W.; Lin, Weiyao

doi:10.1109/icip.2019.8802965

Cited by 95 publications

(45 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This work will contribute to the further exploration of sparse binary descriptors, which will improve the prediction of MEs, and thus combining the advantages of handcrafted features with deep learning technology will be our future work. ELBPTOP * [42] Extended Local Binary Patterns on Three Orthogonal Planes Bi-WOOF+Phase * [43] Bi-Weighted Oriented Optical Flow with phase information ELRCN * [24] Enriched Long-term Recurrent Convolutional Network 3D flow-based CNN * [26] 3D flow-based convolutional neural networks TSCNN * [28] Transferring Long-term Convolutional Neural Network SSSN * [44] Single-Stream Shallow Network DSSN * [44] Dual-Stream Shallow Network…”

Section: Discussionmentioning

confidence: 99%

Sparse Spatiotemporal Descriptor for Micro-Expression Recognition Using Enhanced Local Cube Binary Pattern

Cen

Yan

et al. 2020

Sensors

View full text Add to dashboard Cite

As a spontaneous facial expression, a micro-expression can reveal the psychological responses of human beings. Thus, micro-expression recognition can be widely studied and applied for its potentiality in clinical diagnosis, psychological research, and security. However, micro-expression recognition is a formidable challenge due to the short-lived time frame and low-intensity of the facial actions. In this paper, a sparse spatiotemporal descriptor for micro-expression recognition is developed by using the Enhanced Local Cube Binary Pattern (Enhanced LCBP). The proposed Enhanced LCBP is composed of three complementary binary features containing Spatial Difference Local Cube Binary Patterns (Spatial Difference LCBP), Temporal Direction Local Cube Binary Patterns (Temporal Direction LCBP), and Temporal Gradient Local Cube Binary Patterns (Temporal Gradient LCBP). With the application of Enhanced LCBP, it would no longer be a problem to provide binary features with spatiotemporal domain complementarity to capture subtle facial changes. In addition, due to the redundant information existing among the division grids, which affects the ability of descriptors to distinguish micro-expressions, the Multi-Regional Joint Sparse Learning is designed to perform feature selection for the division grids, thus paying more attention to the critical local regions. Finally, the Multi-kernel Support Vector Machine (SVM) is employed to fuse the selected features for the final classification. The proposed method exhibits great advantage and achieves promising results on four spontaneous micro-expression datasets. Through further observation of parameter evaluation and confusion matrix, the sufficiency and effectiveness of the proposed method are proved.

show abstract

Section: Discussionmentioning

confidence: 99%

Sparse Spatiotemporal Descriptor for Micro-Expression Recognition Using Enhanced Local Cube Binary Pattern

Cen

Yan

et al. 2020

Sensors

View full text Add to dashboard Cite

show abstract

“…Similarly, OFF-ApexNet [22] extracted the optical flows of two directions respectively based on the onset frame and apex frame in one sequence and combined these two streams in a fully-connected layer for MER. In [38], a deep model containing two-stream and two-level convolutional layers based on pretrained AlexNet was used to promote OFF-ApexNet and then fine tuned for recognizing micro-expressions. Different from using the pretrained models, STRCN [23], [24] modeled the spatio-temporal changes both in appearance based and geometric based ways, which can train the deep models from scratch by using data augmentation techniques.…”

Section: A Individual-database Mermentioning

confidence: 99%

“…When the model becomes shallower and less layers are included into the end-to-end framework, less receptive fields are considered, limiting the representation ability of the model with shallower architecture. Inspired by multiple-stream deep models [38], [47], we utilize various convolutional operations with different receptive fields in same layer to mimic different receptive fields, rather than using multiple filters with different initial weights in different layers. However, the standard convolutional filter with larger receptive field implies more learnable parameters, which is not helpful for improving the recognition performance.…”

Section: Parameter-free Extension Modulesmentioning

confidence: 99%

“…5) The Impact of Motion Magnification: As the motion magnification procedure has shown the performance promotion for deep models in the individual-database task [20], [23], [24], [26], [38], we also investigate the influence of motion magnification in the composite-database task. By adding data processing with a motion magnification (MM) procedure [49], the proposed models using basic feature maps and pooling sizes are compared with or without MM procedure.…”

Section: Parameter Analysis and Ablation Studymentioning

confidence: 99%

“…The LBP-TOP was used as the baseline of MEGC2019 and its results were reported on the official website. The results of Bi-WOOF were reported under the same protocol and experimental settings in [25], which used TV-L1 optical flow [38] to compute the optical flow maps and did not use the UAR metric.…”

Section: Comparison To State Of the Artsmentioning

confidence: 99%

See 2 more Smart Citations

Revealing the Invisible With Model and Data Shrinking for Composite-Database Micro-Expression Recognition

Xia

Peng

Khor

et al. 2020

IEEE Trans. on Image Process.

Self Cite

103

View full text Add to dashboard Cite

Composite-database micro-expression recognition is attracting increasing attention as it is more practical to realworld applications. Though the composite database provides more sample diversity for learning good representation models, the important subtle dynamics are prone to disappearing in the domain shift such that the models greatly degrade their performance, especially for deep models. In this paper, we analyze the influence of learning complexity, including the input complexity and model complexity, and discover that the lower-resolution input data and shallower-architecture model are helpful to ease the degradation of deep models in composite-database task. Based on this, we propose a recurrent convolutional network (RCN) to explore the shallower-architecture and lower-resolution input data, shrinking model and input complexities simultaneously. Furthermore, we develop three parameter-free modules (i.e., wide expansion, shortcut connection and attention unit) to integrate with RCN without increasing any learnable parameters. These three modules can enhance the representation ability in various perspectives while preserving not-very-deep architecture for lower-resolution data. Besides, three modules can further be combined by an automatic strategy (a neural architecture search strategy) and the searched architecture becomes more robust. Extensive experiments on MEGC2019 dataset (composited of existing SMIC, CASME II and SAMM datasets) have verified the influence of learning complexity and shown that RCNs with three modules and the searched combination outperform the state-ofthe-art approaches.

show abstract

Research on Micro-expression Recognition Algorithm Based on Graph Convolutional Networks

Shi

Wang

et al. 2022

Advances in Natural Computation, Fuzzy Systems and Knowledge Discovery

View full text Add to dashboard Cite

Dual-stream Shallow Networks for Facial Micro-expression Recognition

Cited by 95 publications

References 19 publications

Sparse Spatiotemporal Descriptor for Micro-Expression Recognition Using Enhanced Local Cube Binary Pattern

Sparse Spatiotemporal Descriptor for Micro-Expression Recognition Using Enhanced Local Cube Binary Pattern

Revealing the Invisible With Model and Data Shrinking for Composite-Database Micro-Expression Recognition

Research on Micro-expression Recognition Algorithm Based on Graph Convolutional Networks

Contact Info

Product

Resources

About