This paper presents a novel feature-based multi-modal image registration technique called selfsimilarity and symmetry with a scale invariant feature transform (3S-SIFT). The proposed technique has the following two main components. First, a ubiquitous problem existing in registering multi-modal images, gradient reversal, is well studied and addressed. Second, the proposed technique takes into account selfsimilarity information between keypoint triangles, which is conducive to enhancing the registration accuracy. Moreover, a simplified version of 3S-SIFT called 4S-SIFT is proposed as a pruning technique for feature matching. The proposed techniques are generally applicable to the registration of multi-modal images with changes in scale, rotation, and translation. The experiments have been conducted on a set of benchmark datasets in the domain of image registration, demonstrating that the proposed techniques achieve the stateof-the-art performance in both matching accuracy and recall. INDEX TERMS Multi-modal image registration, local descriptors, self-similarity, symmetric SIFT.