International Variation in Histologic Grading Is Large, and Persistent Feedback Does Not Improve Reproducibility

Abstract

Histologic grading systems are used to guide diagnosis, therapy, and audit on an international basis. The reproducibility of grading systems is usually tested within small groups of pathologists who have previously worked or trained together. This may underestimate the international variation of scoring systems. We therefore evaluated the reproducibility of an established system, the Banff classification of renal allograft pathology, throughout Europe. We also sought to improve reproducibility by providing individual feedback after each of 14 small groups of cases. Kappa values for all features studied were lower than any previously published, confirming that international variation is greater than interobserver variation as previously assessed. A prolonged attempt to improve reproducibility, using numeric or graphical feedback, failed to produce any detectable improvement. We then asked participants to grade selected photographs, to eliminate variation induced by pathologists viewing different areas of the slide. This produced improved kappa values only for some features. Improvement was influenced by the nature of the grade definitions. Definitions based on "area affected" by a process were not improved. The results indicate the danger of basing decisions on grading systems that may be applied very differently in different institutions.

The Revisited Classification of GN in SLE at 10 Years

“…The reported interobserver agreement for visual assessment of tubular atrophy and interstitial fibrosis using routine stains applied in nephropathology is quite variable. 7,35,36 The reproducible approach reported in the Oxford IgAN classification, 37 in which tubular atrophy and interstitial fibrosis are combined into one grading system, therefore seems most practical.…”

Potential Impact of Microarray Diagnosis of T Cell–Mediated Rejection in Kidney Transplants: The INTERCOM Study

“…Many biopsies are assigned the ambiguous ''borderline'' designation: In our recent study of 403 biopsies, 35 biopsies were diagnosed TCMR but 40 were called borderline (11). Interobserver agreement is poor (12,13), and certain diagnostic rules may be incorrect, for example the designation of all ''isolated v-lesions'' as TCMR (11,14). Central histology assessment is not the solution: It is simply a second opinion using the same standard.…”

“…This reflects the problem of ''reference standard-related bias'': a new test, whether based on a molecular classifier or the opinion of a second pathologist, will not agree perfectly with the existing gold standard (the opinion of a different pathologist) when there is a high degree of subjectivity involved in the assignment of the gold standard. ''Noise'' within histology is intrinsic, due to the poor kappa values for TCMR lesions, for example, tubulitis 0.17, where complete agreement is 1.0 and random is 0.0 (12,13). For the present, the clearest value of the TCMR score will be in problematic cases, and its limited ability to provide information about diagnoses such as recurrent diseases must be acknowledged.…”

How much reproducibility do we need in human and veterinary pathology?

“…However, this is too simple an assumption in view of other investigations that report significant intra-and inter-observer variation among veterinary pathologists in grading canine cutaneous mast cell tumors (Northrup et al, 2005) or intestinal tissues (Willard et al, 2002). Neither is this problem unique to veterinary pathology; it is discussed in human pathology as well (Furness et al, 2003). Pathology and radiology relying heavily on visual interpretation are the two fields of medicine with the lowest diagnostic error rate reproducibility of diagnosis (Berner and Graber, 2008).…”