Multi-modal Retinal Image Registration Using a Keypoint-Based Vessel Structure Aligning Network

Sindel, Aline; Hohberger, Bettina; Maier, Andreas; Christlein, Vincent

doi:10.1007/978-3-031-16446-0_11

Cited by 4 publications

(6 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Wang et al [ 28 ] proposed a content-adaptive multimodal retinal image registration method, which adopted pixel-adaptive convolution (PAC) [ 51 ] and style loss [ 33 ] in their vessel segmentation network. In addition to transforming images into the vessel masks, Santarossa et al [ 22 ] and Sindel et al [ 29 ] applied CycleGAN [ 52 ] to transform the images from one modality to the other before extracting features.…”

Section: Related Workmentioning

confidence: 99%

“…SuperGlue (SG) [ 64 ] and LoFTR [ 63 ] are two direct methods for feature detection and description which were proposed more recently. For indirect methods, we selected two methods [ 28 , 29 ] that utilize different transfer methods, including CycleGAN [ 52 ] and vessel segmentation.…”

Section: Experimental Settingsmentioning

confidence: 99%

“…The official pretrained networks were adopted, where the SuperPoint detection threshold was set as 0.015 and the SuperGlue match threshold was set as 0.1. CycleGAN-based [ 29 ]: This method combined a keypoint detection and description network designed for retinal images (i.e., RetinaCraquelureNet [ 66 ]) with SuperGlue. The networks were trained using self-supervised learning on synthetic multimodal images generated by CycleGAN [ 52 ].…”

Section: Experimental Settingsmentioning

confidence: 99%

“…Multimodal retinal image registration has been extensively studied in recent years [ 22 – 29 ]. However, current approaches have primarily utilized the public CF-FA dataset [ 30 ] (color fundus and fluorescein angiography) or private datasets with modalities other than EMA and OCTA, such as CF and fundus autofluorescence (FAF) [ 22 , 23 ], and CF and infrared reflectance (IR) imaging [ 23 , 27 , 28 ].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

MEMO: dataset and methods for robust multimodal retinal image registration with large or small vessel density differences

Wang,

Sadrieh,

Shen

et al. 2024

Biomed. Opt. Express

View full text Add to dashboard Cite

The measurement of retinal blood flow (RBF) in capillaries can provide a powerful biomarker for the early diagnosis and treatment of ocular diseases. However, no single modality can determine capillary flowrates with high precision. Combining erythrocyte-mediated angiography (EMA) with optical coherence tomography angiography (OCTA) has the potential to achieve this goal, as EMA can measure the absolute RBF of retinal microvasculature and OCTA can provide the structural images of capillaries. However, multimodal retinal image registration between these two modalities remains largely unexplored. To fill this gap, we establish MEMO, the first public multimodal EMA and OCTA retinal image dataset. A unique challenge in multimodal retinal image registration between these modalities is the relatively large difference in vessel density (VD). To address this challenge, we propose a segmentation-based deep-learning framework (VDD-Reg), which provides robust results despite differences in vessel density. VDD-Reg consists of a vessel segmentation module and a registration module. To train the vessel segmentation module, we further designed a two-stage semi-supervised learning framework (LVD-Seg) combining supervised and unsupervised losses. We demonstrate that VDD-Reg outperforms existing methods quantitatively and qualitatively for cases of both small VD differences (using the CF-FA dataset) and large VD differences (using our MEMO dataset). Moreover, VDD-Reg requires as few as three annotated vessel segmentation masks to maintain its accuracy, demonstrating its feasibility.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Experimental Settingsmentioning

confidence: 99%

Section: Experimental Settingsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

MEMO: dataset and methods for robust multimodal retinal image registration with large or small vessel density differences

Wang,

Sadrieh,

Shen

et al. 2024

Biomed. Opt. Express

View full text Add to dashboard Cite

show abstract

“…Sindel et al [ 51 ] used a two-headed network capable of detecting keypoints and creating their cross-modal descriptors. This network is joined with SuperGlue [ 53 ], a graph-neural network capable of point matching, to create an end-to-end training with losses dedicated to the keypoints, the descriptors and their matching.…”

Section: Related Workmentioning

confidence: 99%

Automated inter-device 3D OCT image registration using deep learning and retinal layer segmentation

Rivas-Villar¹,

Motschi²,

Pircher³

et al. 2023

Biomed. Opt. Express

View full text Add to dashboard Cite

Optical coherence tomography (OCT) is the most widely used imaging modality in ophthalmology. There are multiple variations of OCT imaging capable of producing complementary information. Thus, registering these complementary volumes is desirable in order to combine their information. In this work, we propose a novel automated pipeline to register OCT images produced by different devices. This pipeline is based on two steps: a multi-modal 2D en-face registration based on deep learning, and a Z-axis (axial axis) registration based on the retinal layer segmentation. We evaluate our method using data from a Heidelberg Spectralis and an experimental PS-OCT device. The empirical results demonstrated high-quality registrations, with mean errors of approximately 46 µm for the 2D registration and 9.59 µm for the Z-axis registration. These registrations may help in multiple clinical applications such as the validation of layer segmentations among others.

show abstract

A Vesselsegmentation-based CycleGAN for Unpaired Multi-modal Retinal Image Synthesis

Sindel¹,

Maier²,

Christlein³

2023

Informatik Aktuell

View full text Add to dashboard Cite

Multi-modal Retinal Image Registration Using a Keypoint-Based Vessel Structure Aligning Network

Cited by 4 publications

References 23 publications

MEMO: dataset and methods for robust multimodal retinal image registration with large or small vessel density differences

MEMO: dataset and methods for robust multimodal retinal image registration with large or small vessel density differences

Automated inter-device 3D OCT image registration using deep learning and retinal layer segmentation

A Vesselsegmentation-based CycleGAN for Unpaired Multi-modal Retinal Image Synthesis

Contact Info

Product

Resources

About