An improved method of analysis of observer variation between pathologists

Abstract

An improved method of analysing interobserver variation in histopathological studies is described and illustrated, by use of data from a congruence survey of malignant melanoma. The method provides, between any number of pathologists, an assessment of overall agreement and of agreement on each individual category of a classification system. Adjustment for differences in chance agreement due to varying numbers of categories or an altered composition of cases is included in the analysis. A generalization of the procedure designed to measure the strength of associations between different categories is formulated and explained with the use of an example.

Reproducibility of the 1998 World Health Organization/International Society of Urologic Pathology classification of papillary urothelial neoplasms of the urinary bladder

“…Reproducibility was expressed either by the crude rate of agreement or by the kappa (k) statistics [16]. Statistical comparisons in agreement between classifications were made using a paired t-test of differences in k statistics.…”

Agreement in the Clinical Diagnosis of Dementia: Evaluation of a Case Series with Mild Cognitive Impairment

“…This provides a numerical measure of chance-cor rected agreement [14,15]. The formula is as follows: k = (Po -Pe)/(1 -Pe) where Po is the crude proportion of agreement and Pe is the chance-explained propor tion of agreement.…”

“…Since it is a ratio between the chance-corrected observed agreement and the chancecorrected perfect agreement, k approaches +1 for per fect agreement and -1 for complete disagreement; for a completely chance-explained agreement, k equals 0. Statistical significance of k values is tested by dividing it by its standard error, the ratio being distributed as a standard normal value Z [15], x2 analysis was per formed to estimate homogeneity in the attribution of the diagnostic categories by the raters. Table 2 shows the distribution of the diag nostic categories among the raters as well as the original diagnoses made in the hospital.…”

“…This statistic assesses the probability that a patient assigned to the i-th category by one rater will be assigned to any other j-th cat egory by another rater [ 15]. Except for the val ues in italics on the diagonal of All k values are significant at the p < 0.001 level.…”

Reliability and Accuracy in Reporting CIN in 14 Laboratories

“…7 An analysis of exchangeability (related to the probability that a second observer might allocate a case to a particular diagnostic category when the first has indicated a different one) between diagnostic categories was also performed (Table II); for this study a negative kappa meant low interchangeability (a high consistency) of the relevant categories.…”