On Clinical Agreement on the Visibility and Extent of Anatomical Layers in Digital Gonio Photographs

Abstract

Purpose To quantitatively evaluate the inter-annotator variability of clinicians tracing the contours of anatomical layers of the iridocorneal angle on digital gonio photographs, thus providing a baseline for the validation of automated analysis algorithms. Methods Using a software annotation tool on a common set of 20 images, five experienced ophthalmologists highlighted the contours of five anatomical layers of interest: iris root (IR), ciliary body band (CBB), scleral spur (SS), trabecular meshwork (TM), and cornea (C). Inter-annotator variability was assessed by (1) comparing the number of times ophthalmologists delineated each layer in the dataset; (2) quantifying how the consensus area for each layer (i.e., the intersection area of observers’ delineations) varied with the consensus threshold; and (3) calculating agreement among annotators using average per-layer precision, sensitivity, and Dice score. Results The SS showed the largest difference in annotation frequency (31%) and the minimum overall agreement in terms of consensus size (∼28% of the labeled pixels). The average annotator's per-layer statistics showed consistent patterns, with lower agreement on the CBB and SS (average Dice score ranges of 0.61–0.7 and 0.73–0.78, respectively) and better agreement on the IR, TM, and C (average Dice score ranges of 0.97–0.98, 0.84–0.9, and 0.93–0.96, respectively). Conclusions There was considerable inter-annotator variation in identifying contours of some anatomical layers in digital gonio photographs. Our pilot indicates that agreement was best on IR, TM, and C but poorer for CBB and SS. Translational Relevance This study provides a comprehensive description of inter-annotator agreement on digital gonio photographs segmentation as a baseline for validating deep learning models for automated gonioscopy.

Semantic segmentation of gonio-photographs via adaptive ROI localisation and uncertainty estimation

“…Comparing average per-layer metrics on the hold-out test set (table 2) between the two models, our new one returned overall higher Dice scores, in particular for CBB and SS (+2.4% and +5.6% on average), two layers of fundamental clinical importance. To interpret these values correctly, one must consider that, importantly, the average inter-observer Dice scores for CBB and SS reported in our pilot inter-annotator variability study 17 were about 75% and 65% (the lowest), making our results consistent with the average agreement between experts. Our previous approach promoted sensitivity (less false negatives) for layers with lower inter-annotator agreement (CBB and SS) while the new one promotes precision (less false positives).…”

“…In a previous study, we have reported a detailed analysis on the inter-annotator variability of anatomical layers delineations in digital gonio-photographs. 17 In essence, we observed lower average agreement (quantified using Dice scores) for CBB and SS (about 75% and 65%, respectively) than for I, TM and C (about 97%, 87% and 95%, respectively). This will be important when assessing system performance.…”

“…The layer-wise Dice scores on the test set correlate well with results from a pilot inter-annotator variability study. 17 Finally, we discuss the current limitations of our research. We acknowledge the limited amount of data used.…”

Moving beyond the Slit-Lamp Gonioscopy: Challenges and Future Opportunities

“…Digital gonioscopy has been shown useful in documenting post-operative and abnormal findings of the ACA and has a great potential for teaching gonioscopy. Deep learning algorithms for automatic classification of the angle are on the horizon and their validity in the clinical setting should be accurately evaluated [31][32][33]. Informed Consent Statement: Written informed consent has been obtained from the patients to publish images in this paper.…”

“…Peroni et al recently better characterized the agreement on identification of the ACA structures between graders. Using a software annotation tool that permitted tracing the contours of anatomical layers, Peroni et al found that scleral spur had the minimum agreement whereas it was best on iris root, trabecular meshwork and cornea [ 31 ]. Based on these studies, deep learning semantic segmentation algorithms for processing images acquired by GS-1 are currently in development [ 32 , 33 , 34 ].…”