Reliability of Trachoma Clinical Grading—Assessing Grading of Marginal Cases

Abstract

BackgroundClinical examination of trachoma is used to justify intervention in trachoma-endemic regions. Currently, field graders are certified by determining their concordance with experienced graders using the kappa statistic. Unfortunately, trachoma grading can be highly variable and there are cases where even expert graders disagree (borderline/marginal cases). Prior work has shown that inclusion of borderline cases tends to reduce apparent agreement, as measured by kappa. Here, we confirm those results and assess performance of trainees on these borderline cases by calculating their reliability error, a measure derived from the decomposition of the Brier score.Methods and FindingsWe trained 18 field graders using 200 conjunctival photographs from a community-randomized trial in Niger and assessed inter-grader agreement using kappa as well as reliability error. Three experienced graders scored each case for the presence or absence of trachomatous inflammation - follicular (TF) and trachomatous inflammation - intense (TI). A consensus grade for each case was defined as the one given by a majority of experienced graders. We classified cases into a unanimous subset if all 3 experienced graders gave the same grade. For both TF and TI grades, the mean kappa for trainees was higher on the unanimous subset; inclusion of borderline cases reduced apparent agreement by 15.7% for TF and 12.4% for TI. When we assessed the breakdown of the reliability error, we found that our trainees tended to over-call TF grades and under-call TI grades, especially in borderline cases.ConclusionsThe kappa statistic is widely used for certifying trachoma field graders. Exclusion of borderline cases, which even experienced graders disagree on, increases apparent agreement with the kappa statistic. Graders may agree less when exposed to the full spectrum of disease. Reliability error allows for the assessment of these borderline cases and can be used to refine an individual trainee's grading.

Photographic grading for trachoma diagnosis within trachoma impact surveys in Amhara region, Ethiopia

“…When the number of follicles is close to the definitional cut-point (five or more follicles at least 0.5 mm in diameter), it is likely that even experts would disagree on the diagnosis. 4 The increased grading variability around programmatic thresholds and the reliance on a subjective clinical sign as a key programmatic indicator may contribute to the increased number of pre-validation surveillance surveys that return unfavorable results above the 5% threshold. 7 …”

“…Conjunctival photographs have been increasingly utilized as part of trachoma research to validate the quality of field grading assessments and to train field graders. 4–6 However, more data are needed on the feasibility of including photographic grading in large-scale trachoma surveys as part of programmatic monitoring. This study aimed to assess the feasibility of taking photographs alongside field grading, the effectiveness of an in-country grading center and the comparability of field and photographic grading within standard programmatic trachoma surveys in the Amhara region of Ethiopia.…”

Evaluation of photography using head-mounted display technology (ICAPS) for district Trachoma surveys

“…One study comparing images taken with an iPhone 4 to field grading reported low sensitivity (41%) but high specificity (91%) for TF [ 8 ]. Another study evaluated an external attachment which allows the smartphone to take magnified images with greater image pixel-density dedicated to the eyelid [ 9 ]. It also used an external magnification attachment with supplementary light sources and a iPhone 4S smartphone, and reported 84.1% sensitivity and 97.6% specificity for TF [ 7 ].…”

“…We are working to understand the possible sources of mismatch between field and image grades. In the future, it would be interesting to add more image and field graders, as was done in a study in Ethiopia or use a different analytic approach to determine the ground truth of the grade for each child, such as latent class analyses or a binned approach [ 9 , 11 ].…”