An end-to-end deep learning approach for landmark detection and matching in medical images

Abstract

Anatomical landmark correspondences in medical images can provide additional guidance information for the alignment of two images, which, in turn, is crucial for many medical applications. However, manual landmark annotation is labor-intensive. Therefore, we propose an end-to-end deep learning approach to automatically detect landmark correspondences in pairs of two-dimensional (2D) images. Our approach consists of a Siamese neural network, which is trained to identify salient locations in images as landmarks and predict matching probabilities for landmark pairs from two different images. We trained our approach on 2D transverse slices from 168 lower abdominal Computed Tomography (CT) scans. We tested the approach on 22,206 pairs of 2D slices with varying levels of intensity, affine, and elastic transformations. The proposed approach finds an average of 639, 466, and 370 landmark matches per image pair for intensity, affine, and elastic transformations, respectively, with spatial matching errors of at most 1 mm. Further, more than 99% of the landmark pairs are within a spatial matching error of 2 mm, 4 mm, and 8 mm for image pairs with intensity, affine, and elastic transformations, respectively. To investigate the utility of our developed approach in a clinical setting, we also tested our approach on pairs of transverse slices selected from follow-up CT scans of three patients. Visual inspection of the results revealed landmark matches in both bony anatomical regions as well as in soft tissues lacking prominent intensity gradients.

Advanced Meta-Heuristics, Convolutional Neural Networks, and Feature Selectors for Efficient COVID-19 X-Ray Chest Image Classification

“…Large datasets availability and the recent advances in deep learning models have led to the possession of power-assisted algorithms, which beats the medical professionals in various clinical image resolution. These images are such as cancer classification [34] , detection of arrhythmia [35] , [36] , identification of haemorrhage [37] , and diagnosis/detection of diabetic retinopathy [38] . Using radiography, the automated diagnosis of chest diseases has gained a lot of enthusiasm and interest.…”

Automatic Landmarks Correspondence Detection in Medical Images with an Application to Deformable Image Registration

“…The data was transferred in anonymized form through a data transfer agreement. A subset of these scans was the same as used in a previous study (Grewal et al, 2020). DV F large and DV F small were additively applied to the target image to generate the source image with elastic transformation.…”

“…The present approach is an extension of our approach (Grewal et al, 2020) for finding landmark correspondences in 2D CT scan slices. Briefly, the approach proposed in (Grewal et al, 2020) consists of a Siamese network with three modules: two CNN branches with shared weights, a sampling layer, and a descriptor matching module. The CNN branches comprise an image-to-image translation network that maps an input image to a feature map.…”

“…Therefore, there is a strong motivation to replace the entire pipeline for automatically finding landmark correspondences by a neural network. While some deep CNN methods have been developed for automatic landmark detection in medical images (Tuysuzoglu et al, 2018;Ghesu et al, 2016;Grewal et al, 2020), these methods are limited to either 2D datasets or supervised learning of a few manually annotated landmarks. In this study, we present a deep CNN (referred to as "DCNN-Match") for automatic landmarks correspondence detection (i.e., simultaneous landmark detection as well as matching) in 3D images.…”

“…In this study, we present a deep CNN (referred to as "DCNN-Match") for automatic landmarks correspondence detection (i.e., simultaneous landmark detection as well as matching) in 3D images. The presented approach is an extension of our approach for 2D images Grewal et al (2020). Briefly, the neural network is trained on pairs of 3D lower abdominal Computed Tomography (CT) scans such that the network learns to predict landmark locations in both the images along with the correspondence score of each landmark location.…”