Reliability of Ophthalmic Diagnoses in an Epidemiologic Survey1

Abstract

In the Nepal Blindness Survey, 39,887 people in 105 sites were examined by 10 ophthalmologists from Nepal and four other countries during 1981. Ophthalmic protocols were pretested on approximately 3000 subjects; however, interobserver variability was inevitable. To quantify the amount of variability and assess the reliability of important ophthalmic measures, a study of interobserver agreement was conducted. Five ophthalmologists, randomly assigned to one of two examining stations in a single survey site, carried out 529 pairs of examinations. Eighty demographic and ophthalmic variables were assessed at each station. In 62 of 80 (77.5%) measures, observer agreement exceeded 90%. Since pathologic findings were rare, however, chance agreement alone could yield misleadingly high per cent agreement; therefore, the kappa statistic was used for assessing comparative reliability of ophthalmic measures. There were 74 measures for which kappa could be computed and ranked by strength of agreement: 20 (27%) showed excellent agreement (kappa = 0.75-1.00), 39 (53%) showed fair to good agreement (kappa = 0.40-0.74), and 15 (20%) showed poor agreement (kappa less than 0.40). In general, measures dealing with blindness prevalence or causes of blindness showed substantial or almost perfect agreement, while polychotomous descriptions of rare clinical signs demonstrated less agreement.

How reliable is the clinical exam in detecting ocular chlamydial infection?

“…The substantial inter-observer variation among examiners reflects the subjective nature of the clinical exam ( Table 2). The level of observer experience, 10,11 complexity in grading scheme, 12 and possible diagnostic drift in individual observers 11 are factors that have influenced the interobserver agreement in past studies. However, the more objective PCR assay shows agreement between assays (k = 0.98) of a different order than has ever been found with the clinical exam.…”

Catatrac: a novel red light-emitting diode device for screening cataracts in the developing world

“…[11][12][13] Equipment such as slit lamps have been used, as well as more basic equipment such as torches, and  2.5 loupes. 8,14 The availability of an accurate and cheap screening tool that could be easily used by non-ophthalmologists would potentially be of great use in cataract detection.…”

A Weighted Kappa Coefficient for Three Observers as a Measure for Reliability of Expert Ratings on Characteristics in Handball Throwing Patterns

“…Finally, in 1971, Kappa was modified by Fleiss to allow the reproducibility's measurement in cases where several (more than two) observers are judging cases. During the last decades, several papers have appeared in the medical and psychological literature, which discuss Kappa indexes for measuring agreement between two or more raters (Brilliant, Lepowski, & Musch, 1983;Kraemer, 1992;Krummenauer, 2000;Little, Worthingham-Roberts, & Mann, 1984;Posner, Sampson, Caplan, Ward, & Cheney, 1990;Robert & McNaemee, 1998). However, the Fleiss index for multiple raters is unweighted (as the original Kappa presented by Cohen [1960]); hence, it treats all disagreements equally.…”