An international comparison of retinopathy of prematurity grading performance within the Benefits of Oxygen Saturation Targeting II trials

Abstract

PurposeTo investigate whether the observed international differences in retinopathy of prematurity (ROP) treatment rates within the Benefits of Oxygen Saturation Targeting (BOOST) II trials might have been caused by international variation in ROP disease grading.MethodsGroups of BOOST II trial ophthalmologists in UK, Australia, and New Zealand (ANZ), and an international reference group (INT) used a web based system to grade a selection of RetCam images of ROP acquired during the BOOST II UK trial. Rates of decisions to treat, plus disease grading, ROP stage grading, ROP zone grading, inter-observer variation within groups and intra-observer variation within groups were measured.ResultsForty-two eye examinations were graded. UK ophthalmologists diagnosed treat-requiring ROP more frequently than ANZ ophthalmologists, 13.9 (3.49) compared to 9.4 (4.46) eye examinations, P=0.038. UK ophthalmologists diagnosed plus disease more frequently than ANZ ophthalmologists, 14.1 (6.23) compared to 8.5 (3.24) eye examinations, P=0.021. ANZ ophthalmologists diagnosed stage 2 ROP more frequently than UK ophthalmologists, 20.2 (5.8) compared to 12.7 (7.1) eye examinations, P=0.026. There were no other significant differences in the grading of ROP stage or zone. Inter-observer variation was higher within the UK group than within the ANZ group. Intra-observer variation was low in both groups.ConclusionsWe have found evidence of international variation in the diagnosis of treatment-requiring ROP. Improved standardisation of the diagnosis of treatment-requiring ROP is required. Measures might include improved training in the grading of ROP, using an international approach, and further development of ROP image analysis software.

Deep learning in ophthalmology: The technical and clinical considerations

“…[110][111][112][113] In particular regarding clinical care, there are a number of real-world challenges regarding plus disease diagnosis: (1) There is often significant variability in diagnostic classification (plus vs. pre-plus vs. normal), even among experts, [114][115][116][117][118] leading to inconsistent application of evidence-based practice. 119 This has occurred even in NIH-funded multicenter trials. For example, in the CRYO-ROP protocol, confirmation of threshold disease was required by a second unmasked certified examiner performing dilated ophthalmoscopy.…”

“…There is evidence that this causes bias and inconsistency in diagnosis. 126 (5) Studies have shown that there is geographical variation in plus disease diagnosis possibly related to differences in training, 119 and that there may be chronological drift showing a tendency to diagnose "plus disease" more frequently than in the past. 127 (6) The multicenter Supplemental Therapeutic Oxygen for Prethreshold ROP (STOP-ROP) study defined that plus disease is present if there is sufficient venous dilation and arterial tortuosity in at least 2 quadrants, and this definition was incorporated into the 2005 revised ICROP.…”

Evaluation of a deep learning image assessment system for detecting severe retinopathy of prematurity

“…It has been shown that inter-examiner diagnostic variability is high, even among expert ROP clinicians 1. This variability results in clinically significant differences in outcomes for premature infants 5. Appropriate diagnosis and treatment of ROP reduces the risk of progression, emphasising the importance of accurate and timely diagnosis 4.…”

Practice Guidelines for Ocular Telehealth-Diabetic Retinopathy, Third Edition

“…[228][229][230] In addition, the significant variability that exists between examiners diagnosing ROP using BIO also suggests that the use of remote imaging and computer-based image analysis (CBIA) methods may improve accuracy and consistency of diagnosis of plus disease. 228,[231][232][233][234] A study evaluating ROP image grading by eight ROP experts found that there is poor agreement on the classification of plus disease, despite established international standards. 231 This disagreement suggests that treatment recommendations likely vary among providers and that some infants may be undertreated while others are over treated for ROP.…”