A systematic evaluation of human expert agreement on optical coherence tomography biomarkers using multiple devices

Abstract

Objectives: To assess the agreement in evaluating optical coherence tomography (OCT) variables in the leading macular diseases such as neovascular age-related macular degeneration (nAMD), diabetic macular edema (DME) and retinal vein occlusion (RVO) among OCT-certi ed graders.Methods: SD-OCT volume scans of 356 eyes were graded by seven graders. The grading included presence of intra-and subretinal uid (IRF, SRF), pigment epithelial detachment (PED), epiretinal membrane (ERM), conditions of the vitreomacular interface (VMI), central retinal thickness (CRT) at the foveal centerpoint (CP) and central millimeter (CMM), as well as height and location of IRF/SRF/PED. Kappa statistics (κ) and intraclass correlation coe cient (ICC) were used to report categorical grading and measurement agreement.Results: The overall agreement on the presence of IRF/SRF/PED was κ = 0.82/0.85/0.81; κ of VMI condition was 0.77, that of ERM presence 0.37. ICC for CRT measurements at CP and CMM was excellent with an ICC of 1.00. Height measurements of IRF/SRF/PED showed robust consistency with ICC=0.85-0.93. There was substantial to almost perfect agreement in locating IRF/SRF/PED with κ=0.67-0.86.Between diseases, κ of IRF/SRF presence was 0.69/0.80 for nAMD, 0.64/0.83 for DME and 0.86/0.89 for RVO.Conclusion: Even in the optimized setting, featuring certi ed graders, standardized image acquisition and the use of a professional reading platform, there is a disease dependent variability in biomarker evaluation that is most pronounced for IRF in nAMD as well as DME. Our ndings highlight the variability in the performance of human expert OCT grading and the need for AI-based automated feature analyses.

Clinically significant macular edema in an underserved population: Association with demographic factors and hemoglobin A1c

“…50 Differences in agreement depend largely on which criteria are used, and both false-positives and false-negatives are found. 49,50 Grading using hard exudates agrees better with central macular thickness on optical coherence tomography than grading using microaneurysms, but the conclusion that grading exudates from color photography leads to a significant number of false alarms 45 may be unwarranted. 50 Many of the key aspects of diabetic macular edema were not graded, such as hard exudates, various methods for grading photoreceptor changes, disruption of retinal layers, or hyperreflective foci in the fovea as reviewed previously.…”

“…Inner retinal, outer retinal, and subretinal fluids are now included, but exudates are often missing in these analyses. Even in an optimal setting, the agreement among human experts of optical coherence tomography biomarkers is not excellent, with artificial intelligence suggested as an improved way to use the information in the images 49 …”

“…The agreement between classifications based on nonstereo color fundus photographs versus optical coherence tomography varies considerably, as does between different classifications that each use optical coherence tomography 50 . Differences in agreement depend largely on which criteria are used, and both false-positives and false-negatives are found 49,50 . Grading using hard exudates agrees better with central macular thickness on optical coherence tomography than grading using microaneurysms, but the conclusion that grading exudates from color photography leads to a significant number of false alarms 45 may be unwarranted 50 .…”

“…Even in an optimal setting, the agreement among human experts of optical coherence tomography biomarkers is not excellent, with artificial intelligence suggested as an improved way to use the information in the images. 49 The agreement between classifications based on nonstereo color fundus photographs versus optical coherence tomography varies considerably, as does between different classifications that each use optical coherence tomography. 50 Differences in agreement depend largely on which criteria are used, and both false-positives and false-negatives are found.…”

Clinical Characteristics, Treatment Modalities, and Their Association with Long-term Visual Outcomes in Uveal Effusion Syndrome

“…Central macular thickness was measured from the internal limiting membrane to the RPE, and the central SRF height was measured perpendicular to Bruch membrane at the foveal center point. 21 Subfoveal choroidal thickness was measured from the outer surface of the RPE band to the inner surface of the choroidal–scleral interface under the fovea. 22 Posterior scleral thickness was measured on axial B-scans using a caliper within the measurement window of the B-scan device viewing software and was defined as the maximum vertical distance between the retina (the first spike at the back of the eye) and the outer surface of the sclera in the vicinity of the optic nerve head.…”