Analysis of Underlying Causes of Inter-expert Disagreement in Retinopathy of Prematurity Diagnosis

Ataer-Cansızoğlu, Esra; Kalpathy–Cramer, Jayashree; You, Sheng; Keck, Katie; Erdoğmuş, Deni̇z; Chiang, Michael F.

doi:10.3414/me13-01-0081

Cited by 23 publications

(9 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Computer-based image analysis systems may be able to help us better understand the relationships of all of these factors to expert behavior. 11,14,45,46 Third, the implications of these findings are limited to the diagnostic discrepancy in plus disease diagnosis in a single posterior pole image, and do not address other ICROP categories (e.g. zone and stage), or peripheral vascular features, which can influence overall disease severity in the current ICROP classification scheme.…”

Section: Discussionmentioning

confidence: 95%

Plus Disease in Retinopathy of Prematurity

et al. 2016

View full text Add to dashboard Cite

Objective To identify patterns of inter-expert discrepancy in plus disease diagnosis in retinopathy of prematurity (ROP). Design We developed two datasets of clinical images of varying disease severity (100 images and 34 images) as part of the Imaging and Informatics in ROP study, and determined a consensus reference standard diagnosis (RSD) for each image, based on 3 independent image graders and the clinical exam. We recruited 8 expert ROP clinicians to classify these images and compared the distribution of classifications between experts and the RSD. Subjects, Participants, and/or Controls Images obtained during routine ROP screening in neonatal intensive care units. 8 participating experts with >10 years of clinical ROP experience and >5 peer-reviewed ROP publications. Methods, Intervention, or Testing Expert classification of images of plus disease in ROP. Main Outcome Measures Inter-expert agreement (weighted kappa statistic), and agreement and bias on ordinal classification between experts (ANOVA) and the RSD (percent agreement). Results There was variable inter-expert agreement on diagnostic classifications between the 8 experts and the RSD (weighted kappa 0 – 0.75, mean 0.30). RSD agreement ranged from 80 – 94% agreement for the dataset of 100 images, and 29 – 79% for the dataset of 34 images. However, when images were ranked in order of disease severity (by average expert classification), the pattern of expert classification revealed a consistent systematic bias for each expert consistent with unique cut points for the diagnosis of plus disease and pre-plus disease. The two-way ANOVA model suggested a highly significant effect of both image and user on the average score (P<0.05, adjusted R2=0.82 for dataset A, and P< 0.05 and adjusted R2 =0.6615 for dataset B). Conclusions and Relevance There is wide variability in the classification of plus disease by ROP experts, which occurs because experts have different “cut-points” for the amounts of vascular abnormality required for presence of plus and pre-plus disease. This has important implications for research, teaching and patient care for ROP, and suggests that a continuous ROP plus disease severity score may more accurately reflect the behavior of expert ROP clinicians, and may better standardize classification in the future.

show abstract

Section: Discussionmentioning

confidence: 95%

Plus Disease in Retinopathy of Prematurity

et al. 2016

View full text Add to dashboard Cite

show abstract

“…Retinopathy of prematurity (ROP) is an ocular disease observed in infants with low birth weight, especially premature infants. The retinal blood vessels of the infant suffering from ROP develop abnormally [ 1 ], which affects sight and may even lead to blindness in severe cases, e.g., due to retinal detachment. ROP is treatable by laser photocoagulation if the diagnosis and treatment are timely [ 2 ].…”

Section: Introductionmentioning

confidence: 99%

Automated retinopathy of prematurity screening using deep neural networks

et al. 2018

View full text Add to dashboard Cite

Background Retinopathy of prematurity (ROP) is the leading cause of childhood blindness worldwide. Automated ROP detection system is urgent and it appears to be a safe, reliable, and cost-effective complement to human experts. Methods An automated ROP detection system called DeepROP was developed by using Deep Neural Networks (DNNs). ROP detection was divided into ROP identification and grading tasks. Two specific DNN models, i.e., Id-Net and Gr-Net, were designed for identification and grading tasks, respectively. To develop the DNNs, large-scale datasets of retinal fundus images were constructed by labeling the images of ROP screenings by clinical ophthalmologists. Findings On the test dataset, the Id-Net achieved a sensitivity of 96.62%(95%CI, 92.29%–98.89%) and a specificity of 99.32% (95%CI, 96.29%–9.98%) for ROP identification while the Gr-Net attained sensitivity and specificity values of 88.46% (95%CI, 96.29%–99.98%) and 92.31% (95%CI, 81.46%–97.86%), respectively, on the ROP grading task. On another 552 cases, the developed DNNs outperformed some human experts. In a clinical setting, the sensitivity and specificity values of DeepROP for ROP identification were 84.91% (95%CI, 76.65%–91.12%) and 96.90% (95%CI, 95.49%–97.96%), respectively, whereas the corresponding measures for ROP grading were 93.33%(95%CI, 68.05%–99.83%) and 73.63%(95%CI, 68.05%–99.83%), respectively. Interpretation We constructed large-scale ROP datasets with adequate clinical labels and proposed novel DNN models. The DNN models can directly learn ROP features from big data. The developed DeepROP is potential to be an efficient and effective system for automated ROP screening. Fund National Natural Science Foundation of China under Grant 61432012 and U1435213.

show abstract

“…1 That said, numerous studies have shown that a variety of features beyond this standard definition may be considered by physicians while diagnosing plus disease. 12,33,38,43 To address this potential confounder for 12 of the 34 images, 3 experts were asked to re-classify all 34 of the original 2007 cropped images. The average score was not significantly different from their original 2016 gradings (2.60 vs. 2.67); however it was significantly more severe than the original 2007 scores (2.1).…”

Section: Discussionmentioning

confidence: 99%

Plus Disease in Retinopathy of Prematurity: Diagnostic Trends in 2016 Versus 2007

Moleta

Campbell

Kalpathy–Cramer

et al. 2017

American Journal of Ophthalmology

View full text Add to dashboard Cite

Purpose To identify any temporal trends in the diagnosis of plus disease in retinopathy of prematurity (ROP) by experts. Design Reliability analysis Methods ROP experts were recruited in 2007 and 2016 to classify 34 wide-field fundus images of ROP as plus, pre-plus, or normal, coded as “3,” “2,” and “1” respectively in the database. The main outcome was the average calculated score for each image in each cohort. Secondary outcomes included correlation on the relative ordering of the images in 2016 versus 2007, inter-expert agreement, and intra-expert agreement Results The average score for each image was higher for 30/34 (88%) images in 2016 compared to 2007, influenced by fewer images classified as normal (P<0.01), a similar number of pre-plus (P=0.52), and more classified as plus (P<0.01). The mean weighted kappa values in 2006 were 0.36 (range 0.21 – 0.60) compared to 0.22 (range 0 – 0.40) in 2016. There was good correlation between rankings of disease severity between the two cohorts (Spearman’s rank correlation ρ=0.94) indicating near-perfect agreement on relative disease severity. Conclusions Despite good agreement between cohorts on relative disease severity ranking, the higher average score and classifications for each image demonstrate that experts are diagnosing pre-plus and plus disease at earlier stages of disease severity in 2016, compared with 2007. This has implications for patient care, research, and teaching, and additional studies are needed to better understand this temporal trend in image-based plus disease diagnosis.

show abstract

Analysis of Underlying Causes of Inter-expert Disagreement in Retinopathy of Prematurity Diagnosis

Cited by 23 publications

References 33 publications

Plus Disease in Retinopathy of Prematurity

Plus Disease in Retinopathy of Prematurity

Automated retinopathy of prematurity screening using deep neural networks

Plus Disease in Retinopathy of Prematurity: Diagnostic Trends in 2016 Versus 2007

Contact Info

Product

Resources

About