Variability of Grading DR Screening Images among Non-Trained Retina Specialists

Abstract

Poland has never had a widespread diabetic retinopathy (DR) screening program and subsequently has no purpose-trained graders and no established grader training scheme. Herein, we compare the performance and variability of three retinal specialists with no additional DR grading training in assessing images from 335 real-life screening encounters and contrast their performance against IDx-DR, a US Food and Drug Administration (FDA) approved DR screening suite. A total of 1501 fundus images from 670 eyes were assessed by each grader with a final grade on a per-eye level. Unanimous agreement between all graders was achieved for 385 eyes, and 110 patients, out of which 98% had a final grade of no DR. Thirty-six patients had final grades higher than mild DR, out of which only two had no grader disagreements regarding severity. A total of 28 eyes underwent adjudication due to complete grader disagreement. Four patients had discordant grades ranging from no DR to severe DR between the human graders and IDx-DR. Retina specialists achieved kappa scores of 0.52, 0.78, and 0.61. Retina specialists had relatively high grader variability and only a modest concordance with IDx-DR results. Focused training and verification are recommended for any potential DR graders before assessing DR screening images.

Diagnostic Accuracy of Automated Diabetic Retinopathy Image Assessment Softwares: IDx-DR and Medios Artificial Intelligence

“…The possible reasons for variability among graders could be identification and differentiation of subtle DR features (retinal hemorrhages, microaneurysms, hard exudates, new vessels, intraretinal microvascular abnormalities, neovascularization, and surrogate markers of macular edema), variation in image quality due to artifacts, brightness or contrast of images. This has been found in other studies and in other fields of medical imaging as well [ 13 – 15 ]. Gold standard grading in the current study was done based on a majority decision by the graders.…”

“…Inter-grader and intra-grader variability is known among DR graders. Previous studies have shown that inter-grader kappa scores typically range from 0.40 to 0.65 in DR grading [13][14][15][16][17][18][19]. The kappa values for both Polish (0.68-0.90) and Indian graders (0.63-0.91) in the study showed similar variability but were well within the limits showing overall good agreement.…”

“…Using a non-mydriatic protocol may underrepresent patients with smaller pupils or media opacities, particularly the elderly. In a previous study about DR grader reliability, based on the same screening program in Poland from which images for this study were taken, out of 495 patients only 335 were deemed to be of sufficient quality by IDx-DR and all three human graders [ 15 ]. How many of those low-quality screening encounters could we image and diagnose after mydriasis remains to be seen, as is the comparative performance of both systems in those patients.…”

Spotlight on iron and ferroptosis: research progress in diabetic retinopathy

“… Dilation-free funduscopies of DR from ( 122 ). The left image shows multiple small hemorrhages and microaneurysms.…”

Variability in Grading Diabetic Retinopathy Using Retinal Photography and Its Comparison with an Automated Deep Learning Diabetic Retinopathy Screening Software

“…This study analysed locally acquired colour fundus photographs at a tertiary centre in Singapore and revealed a significant range of intra-group and inter-observer variability among human graders with different levels of expertise, particularly in the non-ophthalmology group. Notably, a previous study conducted by Grzybowski A et al demonstrated high variability in diabetic retinopathy grading among non-trained retina specialists [ 25 ]. These findings underscore the need for global improvements in the current grading of diabetic retinopathy severity [ 25 ], especially in centres where human graders are predominantly involved.…”