A novel deep learning conditional generative adversarial network for producing angiography images from retinal fundus photographs

Abstract

Fluorescein angiography (FA) is a procedure used to image the vascular structure of the retina and requires the insertion of an exogenous dye with potential adverse side effects. Currently, there is only one alternative non-invasive system based on Optical coherence tomography (OCT) technology, called OCT angiography (OCTA), capable of visualizing retina vasculature. However, due to its cost and limited view, OCTA technology is not widely used. Retinal fundus photography is a safe imaging technique used for capturing the overall structure of the retina. In order to visualize retinal vasculature without the need for FA and in a cost-effective, non-invasive, and accurate manner, we propose a deep learning conditional generative adversarial network (GAN) capable of producing FA images from fundus photographs. The proposed GAN produces anatomically accurate angiograms, with similar fidelity to FA images, and significantly outperforms two other state-of-the-art generative algorithms ($$p<.001$$ p < . 001 and $$p<.0001$$ p < . 0001 ). Furthermore, evaluations by experts shows that our proposed model produces such high quality FA images that are indistinguishable from real angiograms. Our model as the first application of artificial intelligence and deep learning to medical image translation, by employing a theoretical framework capable of establishing a shared feature-space between two domains (i.e. funduscopy and fluorescein angiography) provides an unrivaled way for the translation of images from one domain to the other.

Application of generative adversarial networks (GAN) for ophthalmology image domains: a survey

“…c Super-resolution for optic nerve head photography [ 56 ]. d Domain transfer for fundus photography to angiography [ 62 ] and ultra-widefield to classic fundus photography (re-analysis in this work) [ 63 ]. e Data augmentation for ocular surface images [ 46 ] and anterior segment OCT [ 82 ].…”

“…Autoencoder was used to synthesize new retinal vessel images apart from training of GAN Zhao et al [ 59 ] Conditional GAN Vessel image → Fundus photography Retinal image synthesis can be effectively learned in a data-driven fashion from a relatively small sample size using a conditional GAN architecture Yu et al [ 60 ] Pix2pix (with ResU-net generator) (conditional GAN) Vessel image → Fundus photography To enlarge training datasets for facilitating medical image analysis, the multiple-channels-multiple-landmarks (MCML) was developed to synthesize color fundus images from a combination of vessel and optic disc masked images Wu et al [ 61 ] Conditional GAN Volumetric retinal OCT → Fundus autofluorescence The en-face OCT images were synthesized from volumetric retinal OCT by restricted summed voxel projection. The fundus autofluorescence images were generated from en-face OCT images using GAN to identify the geographic atrophy region Tavakkoli et al [ 62 ] Conditional GAN Fundus photography → Fluorescein angiography The proposed GAN produced anatomically accurate fluorescein angiography images that were indistinguishable from real angiograms Yoo et al [ 63 ] CycleGAN Ultra-widefield fundus photography → Fundus photography Ultra-widefield images were successfully translated into traditional fundus photography-style images by CycleGAN, and the main structural information of the retina and optic nerve was retained Ju et al [ 64 ] CycleGAN Fundus photography → Ultra-widefield fundus photography The CycleGAN model transferred the color fundus photographs to ultra-widefield images to introduce additional data for existing limited ultra-widefield images. The proposed method was adopted for diabetic retinopathy grading and lesion detection Lazaridis et al [ 91 , 108 ] Wasserstein GAN + perceptual loss (conditional GAN) Time-domain OCT → spectral-domain OCT Time-domain OCT was converted to synthetic spectral-domain OCT using GAN.…”

“…In that study, en-face OCT images from volumetric OCT data were successfully transformed into synthetic autofluorescence images to detect geographic atrophy regions in the retina. Tavakkoli et al demonstrated that realistic retinal angiography images with diabetic retinopathy were generated via conditional GAN using fundus photographs [ 62 ]. Although the model was trained using a limited angiography dataset without detailed phase information, the study showed the potential of image domain transfer for the diagnosis of diabetic retinopathy.…”

Deepfakes in Ophthalmology

“…We found 20 published implementations of GANs specific to ophthalmology. Of these, 11 manuscripts synthesized fundus images, 9,19,20,23,32,37,[40][41][42][43][44] 6 manuscripts synthesized OCT images, [27][28][29][45][46][47] 2 manuscripts synthesized fluorescein angiography images, 48,49 and 1 manuscript synthesized infrared images 21 (Table 2). The majority of GANs were proof-ofconcept studies demonstrating feasibility of generating realistic-appearing synthetic images, specific implementations of GANs were published in 9 for diagnosis of ophthalmic diseases, including diabetic retinopathy (DR), 9,20,32,40 glaucoma, 28,45 age-related macular degeneration (AMD), 19,46 and meibomian gland dysfunction.…”

RF‐GANs: A Method to Synthesize Retinal Fundus Images Based on Generative Adversarial Network

“…But in several cases, as the artifact became more obscure, the checkerboard-like artifact became more prominent in generated images. Tavakkoli et al [ 34 ] used a two-stage generator based on CGAN and used retinal fundus images as inputs for fluorescein angiography (FA) generation. The method could produce high-quality FA images even when the quality of their counterparts was relatively low.…”