Intereye Agreement of Retinopathy of Prematurity from Image Evaluation in the Telemedicine Approaches to Evaluating of Acute-Phase ROP (e-ROP) Study

Abstract

Purpose To determine the symmetry on retinal image grading of fellow eyes for retinopathy of prematurity (ROP) features (stage, zone and plus disease) and severity, to provide the basis for the within subject comparison for ROP trials. Design Secondary analyses of data from the Telemedicine Approaches to Evaluating of Acute-Phase ROP (e-ROP) Study. Subjects Infants with birth weight less than 1251g. Methods Infants underwent serial retinal imaging sessions in both eyes by certified imagers starting at 32-weeks postmenstrual age (PMA). Two trained non-physician readers graded each eye independently for ROP features in a 5 retinal-image set from each session. Discrepancies were adjudicated by a reading center supervisor. Readers were masked to all eye examination results, previous gradings of both eyes, current grading of the fellow eye, and demographic data. Main Outcome Measures The inter-eye agreement assessed using percent exact agreement and weighted kappa (Kw) for stage, zone, plus, referral-warranted ROP (RW-ROP, defined as presence of stage 3 or above, plus disease, or zone I ROP), and severity of ROP. Results Among 3918 image sessions in 1235 infants, the percent agreement (Kw) between paired eyes was 75.3% (0.65) for stage of ROP, 82.3% (0.68) for zone of ROP, 78.7% (0.51) for plus disease, 84.7% (0.56) for RW-ROP, and 72.7% (0.63) for severity of ROP. Similar inter-eye agreements were found when considering ROP features at the first image session, at the last image session, at any image session. Based on image evaluations from all sessions, 412 (33.4%) infants had ROP stage 3 or above, 148 (12.0%) had zone I ROP, 70 (5.7%) had plus disease, and 419 (33.9%) had RW-ROP in one or both eyes; symmetrical findings were present in 71.4% for ROP stage 3 or above, 56.8% for zone I ROP, 50.0% for plus disease, and 73.7% for RW-ROP. Conclusions Masked image evaluations by trained readers showed good inter-eye agreement in ROP characteristics, consistent with the high inter-eye agreement in ROP from clinical examinations by ophthalmologists in other studies. These data confirm that acute ROP is typically symmetrical and supports within subject comparison in ROP trials.

Assessing Intereye Symmetry and Its Implications for Study Design

“…If presence of the feature were perfectly symmetric (i.e., all subjects either have two eyes affected or zero eyes affected), then the proportions of affected left and right eyes would be the same. Ying et al 1 investigated the symmetry of the retinopathy of prematurity (ROP) between eyes of 1180 premature infants and found that 353 (29.9%) right eyes and 364 (30.8%) left eyes had referral-warranted ROP (RW-ROP). These proportions are nearly equal, consistent with having symmetry between eyes; however, the proportions could be equal even if the agreement between right and left eyes was no more than expected by chance; that is, having equal proportions is a necessary but not sufficient requirement for perfect symmetry.…”

“…Newcombe 2 describes several methods for calculating a 95% κ w = 0.86. 95% CI of (0.85, 0.88), calculated using squared error weights and the large sample variance from Fleiss et al 4 Data from table 1 of the publication by Ying et al 1 CI on the difference in proportions for right and left eyes; one or more of these methods are available in most statistical software packages. The Wilson score method (sometimes referred to as the Newcombe score method) and likelihood profile methods are generally preferred to the Wald method.…”

“…Features in eyes may be classified into levels of severity or size that are pertinent in assessing symmetry. Ying et al 1 also classified eyes by stage of ROP, in which stage 0 indicated no ROP (Table 2). Again, a first step is to compare the percentages in each category between right and left eyes (right eye percent, left eye percent).…”

“… κ w = 0.86. 95% CI of (0.85, 0.88), calculated using squared error weights and the large sample variance from Fleiss et al 4 Data from table 1 of the publication by Ying et al 1 …”

“…Cross-Classification of Right and Left Eyes for RW-Data from table 7 of the publication by Ying et al1 …”