Artificial Intelligence for Automated Overlay of Fundus Camera and Scanning Laser Ophthalmoscope Images

Abstract

Purpose The purpose of this study was to evaluate the ability to align two types of retinal images taken on different platforms; color fundus (CF) photographs and infrared scanning laser ophthalmoscope (IR SLO) images using mathematical warping and artificial intelligence (AI). Methods We collected 109 matched pairs of CF and IR SLO images. An AI algorithm utilizing two separate networks was developed. A style transfer network (STN) was used to segment vessel structures. A registration network was used to align the segmented images to each. Neither network used a ground truth dataset. A conventional image warping algorithm was used as a control. Software displayed image pairs as a 5 × 5 checkerboard grid composed of alternating subimages. This technique permitted vessel alignment determination by human observers and 5 masked graders evaluated alignment by the AI and conventional warping in 25 fields for each image. Results Our new AI method was superior to conventional warping at generating vessel alignment as judged by masked human graders ( P < 0.0001). The average number of good/excellent matches increased from 90.5% to 94.4% with AI method. Conclusions AI permitted a more accurate overlay of CF and IR SLO images than conventional mathematical warping. This is a first step toward developing an AI that could allow overlay of all types of fundus images by utilizing vascular landmarks. Translational Relevance The ability to align and overlay imaging data from multiple instruments and manufacturers will permit better analysis of this complex data helping understand disease and predict treatment.

Efficacy and accuracy of artificial intelligence to overlay multimodal images from different optical instruments in patients with retinitis pigmentosa

“…There is a relative lack of papers describing retinal image overlay using AI. The efficacy of AI to overlay retinal images taken by different imaging systems has previously been investigated by Cavichini et al 10 In that study from our group, Cavichini et al overlayed matched pairs of colour fundus (CF) photographs and infrared scanning laser ophthalmoscope (IR SLO) images using conventional alignment methods and AI. The AI overlay strategy used consisted of a joint vessel segmentation and a deformable registration model based on the convolutional neural network (CNN).…”

“…Advances in AI now allow for the combination of multimodal imaging to be used as a co‐localised database. AI has been previously used in a similar way to categorise or detect disease 8–10 . On the other hand, preliminary trials to overlay multimodal imaging, taken on different platforms, using Microsoft PowerPoint, was not successful (Figure 1).…”

Recent Advances and Clinical Application of Color Scanning Laser Ophthalmoscope

“…Recently, there has been a lot of research on artificial intelligence (AI) using fundus photographs, OCT, and OCT angiography images [ 36 , 37 , 38 , 39 , 40 ], and there is a highly likelihood that color SLO, which has high image quality with a large amount of information, will be useful for this kind of automatic diagnosis by AI. Indeed, preliminary studies using AI and color SLO have been reported [ 16 , 41 , 42 ].…”

Color fundus photograph registration based on feature and intensity for longitudinal evaluation of diabetic retinopathy progression

“…Third, some state-of-the-art methods should be compared with the local dataset FI-LORE, but due to the lack of reliable source codes and our inability to completely repeat the methods, we failed to put them into further comparison. Finally, there are some artificial intelligence algorithms that have been developed for retinal image registration tasks [22,47,48]. We have compared one deep learning algorithm, but further studies are needed to investigate deep learning in the context of retinal image alignment.…”