Large sample standard errors of kappa and weighted kappa.

Abstract

Two statistics, kappa and weighted kappa, are available for measuring agreement between two raters on a nominal scale. Formulas for the standard errors of these two statistics have been given in the literature, but they are in error. The errors seem to be in the direction of overestimation, so that the use of the incorrect formulas results in conservative significance tests and confidence intervals. Valid formulas for the approximate large-sample variances are given, and their calculation is illustrated using a numerical example.The statistics kappa (Cohen, 1960) and weighted kappa (Cohen, 1968) were introduced to provide coefficients of agreement between two raters for nominal scales. Kappa is appropriate when all disagreements may be considered equally serious, and weighted kappa is appropriate when the relative seriousness of the different possible disagreements can be specified.The papers describing these two statistics also present expressions for their standard errors. These expressions are incorrect, having been derived from the contradictory assumptions of fixed marginal totals and binomial variation of cell frequencies. Everitt (1968) derived the exact variances of weighted and unweighted kappa when the parameters are zero by assuming a generalized hypergeometric distribution. He found these expressions to be far too complicated for routine use, and offered, as alternatives, expressions derived by assuming binomial distributions. These alternative expressions are incorrect, essentially for the same reason as above.Assume that N subjects are distributed into k* cells by each of them being assigned to one of k categories by one rater and, independently, to one of the same k categories by a second

Global functioning and disability in dissociative disorders

“…Inter-rater reliability for the diagnosis of the five DSM-IV DDs according to the SCID-D-R between the assessors and the first author was calculated as Fleiss kappa (Fleiss et al, 1969) including an adjusted bootstrap confidence interval, based on M = 1000 bootstrap replications, on all subjects who permitted videotaping their SCID-D-R interview (N=84).…”

Statistical Techniques For Comparing Measurers And Methods Of Measurement: A Critical Review

“…For estimating CI, the formulation is (ASE1). A simple method of doing this for K was described by Cohen, 12 a better method by Fleiss et al 14,20 However, for testing the null hypothesis K = 0, a different formulation is required (ASE0). Cohen's original formulation of ASE0 12 has since been improved on by others.…”

“…Cohen's original formulation of ASE0 12 has since been improved on by others. 14,16,20 Different formulations of ASE are needed for evaluating Kw. Again, a simple method was described by Cohen.…”

“…13 Subsequently, more complex but theoretically more accurate methods were described by Fleiss and others. 14,16,20 An empirical approach A very simple method for evaluating agreement for the unweighted kappa statistic was proposed by Landis and Koch. 21 It is as follows: K < 0.00, poor; K = 0.00-0.20, slight; K = 0.21-0.40, fair; K = 0.41-0.60, moderate; K = 0.61-0.80, substantial; K = 0.81-1.00, almost perfect.…”

The Value of Ultrasonography in Hand Surgery

“…Several publications describe the use of ultrasonographic imaging of the hand in posttraumatic and sports-related injuries (13,15,19). Well described are tears of the collateral ligaments of the thumb with the typical Stener lesion (9,14), tendon ruptures, annular pulley lesions in "climber's finger" (17), and the detection of foreign bodies in the soft tissue (11). Sonographic evaluations in rheumatic diseases detect tenosynovitis, synovitis of the wrist and finger joints, and nerve compressions (23).…”