Shallow Optical Flow Three-Stream CNN For Macro- And Micro-Expression Spotting From Long Videos

Abstract

Facial expressions vary from the visible to the subtle. In recent years, the analysis of micro-expressions-a natural occurrence resulting from the suppression of one's true emotions, has drawn the attention of researchers with a broad range of potential applications. However, spotting microexpressions in long videos becomes increasingly challenging when intertwined with normal or macro-expressions. In this paper, we propose a shallow optical flow three-stream CNN (SOFTNet) model to predict a score that captures the likelihood of a frame being in an expression interval. By fashioning the spotting task as a regression problem, we introduce pseudo-labeling to facilitate the learning process. We demonstrate the efficacy and efficiency of the proposed approach on the recent MEGC 2020 benchmark, where state-of-the-art performance is achieved on CAS(ME) 2 with equally promising results on SAMM Long Videos.

YOLOv4‐tiny‐based robust RGB‐D SLAM approach with point and surface feature fusion in complex indoor environments

“…The data set in the indoor environment was collected by the Technical University of Munich, Germany. It has been widely used to evaluate the performance of SLAM algorithms (Liong et al, 2021). The images in this data set were captured using eight high‐precision cameras in a real indoor environment.…”

Lite General Network and MagFace CNN for Micro-Expression Spotting in Long Videos

“…The facial action units involved in one ME are distributed among the eyebrows, mouth, and corners of the eyes [27]. In our paper, three ROIs with an extra border of 4 pixels were extracted: (1) the left eye and left eyebrow, (2) the right eye and right eyebrow, and (3) the mouth.…”

“…In our paper, three ROIs with an extra border of 4 pixels were extracted: (1) the left eye and left eyebrow, (2) the right eye and right eyebrow, and (3) the mouth. We cut the ROIs in accordance with the alignment image, resized regions ( 1) and ( 2) each to a size of 56×56 pixels, stitched them into a single image of 56×112 pixels and combined them with the resized 56×112 pixel image obtained from region (3) to finally obtain a new 112×112 pixel image containing the main facial action units [27,32].…”

“…In the research of previous work [2,15,27], it was proven that optical flow features are advantageous for characterizing subtle spatiotemporal movement information. The TV-L1 method [30] was applied to calculate the optical flow𝑓 between the i-th frame 𝐹 𝑖 and the (i+k)-th frame 𝐹 𝑖+𝑘 ,…”

“…The CNN module has been commonly constructed as the basis for recognizing the changes in facial AUs [20,24,25], and a series of post-processing methods would be combined to obtain the expression intervals [26]. Unlike prevailing work regarding ME spotting as a classification task, Liong et al [27] built a binary search method to automatically locate apex frames and a shallow optical flow threestream CNN to formulate the localization task as a regression problem, which could alleviate the imbalance between positive and negative samples. Although Liong's method effectively captured apex frames of MEs in video sequences, there still exist certain problems.…”