Facial Expression Recognition based on The Fusion of CNN and SIFT Features

Wang, Huibai; Hou, Siyang

doi:10.1109/iceiec49280.2020.9152361

Cited by 8 publications

(2 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The combination of SIFT descriptors along with CNNs has attracted increasing interest recently [ 17 ]. In most of the proposed works, the SIFT features are merged with the CNN features at the final stage just before the classification topology [ 18 , 19 ]. Thus, two streams are utilized independently; on the one hand, is the implementation of the calculation of the SIFT descriptors along with a k-means algorithm for the bag-of-words encoding, and, on the other hand, the CNN features are extracted utilizing a deep learning model.…”

Section: Related Workmentioning

confidence: 99%

“…In this manner, many different approaches are proposed for the calculation of the local descriptors, either exploiting key-point SIFT [ 20 , 21 ] or jointly exploited with dense SIFT features [ 22 ]. Besides, the fusion method is varied from a simple concatenation to more sophisticated attention mechanisms [ 18 , 23 , 24 ]. Additionally, the previous dual-stream logic is modified by redoubling each stream and implementing a Siamese scheme [ 25 ].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

SIFT-CNN: When Convolutional Neural Networks Meet Dense SIFT Descriptors for Image and Sequence Classification

et al. 2022

View full text Add to dashboard Cite

Despite the success of hand-crafted features in computer visioning for many years, nowadays, this has been replaced by end-to-end learnable features that are extracted from deep convolutional neural networks (CNNs). Whilst CNNs can learn robust features directly from image pixels, they require large amounts of samples and extreme augmentations. On the contrary, hand-crafted features, like SIFT, exhibit several interesting properties as they can provide local rotation invariance. In this work, a novel scheme combining the strengths of SIFT descriptors with CNNs, namely SIFT-CNN, is presented. Given a single-channel image, one SIFT descriptor is computed for every pixel, and thus, every pixel is represented as an M-dimensional histogram, which ultimately results in an M-channel image. Thus, the SIFT image is generated from the SIFT descriptors for all the pixels in a single-channel image, while at the same time, the original spatial size is preserved. Next, a CNN is trained to utilize these M-channel images as inputs by operating directly on the multiscale SIFT images with the regular convolution processes. Since these images incorporate spatial relations between the histograms of the SIFT descriptors, the CNN is guided to learn features from local gradient information of images that otherwise can be neglected. In this manner, the SIFT-CNN implicitly acquires a local rotation invariance property, which is desired for problems where local areas within the image can be rotated without affecting the overall classification result of the respective image. Some of these problems refer to indirect immunofluorescence (IIF) cell image classification, ground-based all-sky image-cloud classification and human lip-reading classification. The results for the popular datasets related to the three different aforementioned problems indicate that the proposed SIFT-CNN can improve the performance and surpasses the corresponding CNNs trained directly on pixel values in various challenging tasks due to its robustness in local rotations. Our findings highlight the importance of the input image representation in the overall efficiency of a data-driven system.

show abstract

Section: Related Workmentioning

confidence: 99%