SIFT-CNN: When Convolutional Neural Networks Meet Dense SIFT Descriptors for Image and Sequence Classification

Abstract

Despite the success of hand-crafted features in computer visioning for many years, nowadays, this has been replaced by end-to-end learnable features that are extracted from deep convolutional neural networks (CNNs). Whilst CNNs can learn robust features directly from image pixels, they require large amounts of samples and extreme augmentations. On the contrary, hand-crafted features, like SIFT, exhibit several interesting properties as they can provide local rotation invariance. In this work, a novel scheme combining the strengths of SIFT descriptors with CNNs, namely SIFT-CNN, is presented. Given a single-channel image, one SIFT descriptor is computed for every pixel, and thus, every pixel is represented as an M-dimensional histogram, which ultimately results in an M-channel image. Thus, the SIFT image is generated from the SIFT descriptors for all the pixels in a single-channel image, while at the same time, the original spatial size is preserved. Next, a CNN is trained to utilize these M-channel images as inputs by operating directly on the multiscale SIFT images with the regular convolution processes. Since these images incorporate spatial relations between the histograms of the SIFT descriptors, the CNN is guided to learn features from local gradient information of images that otherwise can be neglected. In this manner, the SIFT-CNN implicitly acquires a local rotation invariance property, which is desired for problems where local areas within the image can be rotated without affecting the overall classification result of the respective image. Some of these problems refer to indirect immunofluorescence (IIF) cell image classification, ground-based all-sky image-cloud classification and human lip-reading classification. The results for the popular datasets related to the three different aforementioned problems indicate that the proposed SIFT-CNN can improve the performance and surpasses the corresponding CNNs trained directly on pixel values in various challenging tasks due to its robustness in local rotations. Our findings highlight the importance of the input image representation in the overall efficiency of a data-driven system.

The Asian Conference on Media, Communication and Film 2022: Official Conference Proceedings

“…SITF-CNN algorithm, proposed by (Tsourounis et al, 2022) is a an algorithm combining both the SIFT and CNN to form one algorithm. It works by feeding the SIFT image representation into CNN.…”

“…It works by feeding the SIFT image representation into CNN. The grey scale image remains an issue with SIFT-CNN algorithm as with CNN algorithm, and is highly suitable for small datasets (Tsourounis et al, 2022).…”

“…The advantages of SIFT-CNN are that it has potential of eliminating the amount of training required by CNN algorithm by using the results from SIFT to train the model and use end-toend learning scheme (Tsourounis et al, 2022).…”

Apple-Net: A Model Based on Improved YOLOv5 to Detect the Apple Leaf Diseases

“…After that, the extracted local features were used as the matching template. The vectorization matrix [ 39 ] of key points in the template diagram is as follows: then we input the image to be detected, carried out gray and binary processing on the detected image, used the SIFT algorithm to extract leaf surface feature points, vectorized the extracted key points, and took the extracted local features as the observation map. The vectorization matrix [ 40 ] of key points in the observation chart is as follows: then the template map and observation map were measured for similarity, and the measurement formula [ 41 ] is as follows: where and represent the eigenvalues of key points in the template matrix and observation matrix, respectively.…”