Coefficient Kappa: Some Uses, Misuses, and Alternatives

Disclosure: The authors have no financial or other potential conflicts of interest.BACKGROUND: Central venous catheter (CVC) insertions are performed frequently by internal medicine residents.Complications, including arterial puncture and pneumothorax, decrease when operators use fewer needle passes to insert the CVC. In this study, we evaluated the effect of simulation-based mastery learning on CVC insertion skill. DESIGN:This was a cohort study of internal jugular (IJ) and subclavian (SC) CVC insertions by 41 internal medicine residents rotating through the medical intensive care unit (MICU) over a five-month period. Thirteen traditionallytrained residents were surveyed about the number of needle passes, complications, and procedural self-confidence on CVCs inserted in the MICU. Concurrently, 28 residents completed simulation-based training in IJ and SC CVC insertions. Simulator-trained residents were expected to perform CVC insertions to mastery standards on a central line simulator. Simulator-trained residents then rotated through the MICU and were surveyed regarding CVC placement. The impact of simulation training was assessed by comparing group survey results. Central venous catheter (CVC) insertions are commonly performed at the bedside in medical intensive care unit (MICU) settings. Internal medicine residents are required to demonstrate knowledge regarding CVC indications, complications, and sterile technique, 1 and often perform the procedure during training. Education in CVC insertion is needed because many internal medicine residents are uncomfortable performing this procedure. 2 CVC insertion also carries the risk of potentially life-threatening complications including infection, pneumothorax, arterial puncture, deep vein thrombosis, and bleeding. Education and training may also contribute to improved patient care because increased physician experience with CVC insertion reduces complication risk. 3,4 Similarly, a higher number of needle passes or attempts during CVC insertion correlates with mechanical complications such as pneumothorax or arterial punctures. 4-8 Pneumothorax rates for internal jugular (IJ) CVCs have been reported to range from 0% to 0.2% and for subclavian (SC) CVCs from 1.5% to 3.1%. 4,5 The arterial puncture rate for IJ CVCs ranges from 5.0% to 9.4% and for SC CVCs from 3.1% to 4.9%. 4,5 Proper use of ultrasound to assist with IJ CVC insertion has been shown to decrease these mechanical complications. 4,5 However, studies of ultrasound use with SC CVC insertion have mixed results. 4 Simulation-based training has been used in medical education to increase knowledge, provide opportunities for deliberate and safe practice, and shape the development of RESULTS: Journal of Hospital Medicine Vol 4 No 7 September 2009 397clinical skills. 9,10 We previously used simulation-based mastery learning to improve the thoracentesis and advanced cardiac life support (ACLS) skills of internal medicine residents. 11,12 Although a few small studies have linked simulation-based interventions to imp...

show abstract

“…27 Within-group differences from pretest (baseline) to posttest (outcome) were analyzed using paired t-tests.…”

Section: Resultsmentioning

confidence: 99%

Use of simulation‐based mastery learning to improve the quality of central venous catheter placement in a medical intensive care unit

McGaghie

Cohen

Balachandran

et al. 2009

Journal of Hospital Medicine

348

272

View full text Add to dashboard Cite

show abstract

“…The kappa coefficient (Cohen, 1960;Fleiss, 1975;Kraemer, 1979;Brennan and Prediger, 1981;Zwick 1988;Warrens, 2008aWarrens, ,b, 2010) is a popular descriptive statistic for summarizing the cross classification of two nominal variables with k ∈ N ≥2 identical categories. These k × k tables occur in various fields of science, including psychometrics, educational measurement, epidemiology, map comparison (Visser and De Nijs, 2006) and content analysis (Krippendorff, 2004).…”

Section: Introductionmentioning

confidence: 99%

Cohen’s kappa can always be increased and decreased by combining categories

Warrens

2010

Statistical Methodology

View full text Add to dashboard Cite

Abstract. The kappa coefficient is a popular descriptive statistic for summarizing the cross classification of two nominal variables with identical categories. It has been frequently observed in the literature that combining two categories increases the value of kappa. In this note we prove the following existence theorem for kappa: For any nontrivial k × k agreement table with k ∈ N ≥3 categories, there exist two categories such that, when combined, the kappa value of the collapsed (k − 1) × (k − 1) agreement table is higher than the original kappa value. In addition, there exist two categories such that, when combined, the kappa value of the collapsed table is smaller than the original kappa value.

show abstract

“…The κ coefficient (Cohen, 1960;Fleiss, 1981;Brennan & Prediger, 1981;Maclure & Willett, 1987;Zwick, 1988;Hsu & Field, 2003;Warrens, 2008aWarrens, ,b,c, 2010aWarrens, ,b,d, 2011) is a popular descriptive statistic for summarizing the cross classification of two nominal variables with n ∈ N ≥2 identical categories. Originally proposed as a measure of agreement between two observers who each rate the same sample of objects (individuals, observations) on a nominal (unordered) scale with the same number of n categories, κ has been applied to square cross-classifications encountered in psychometrics, educational measurement, epidemiology, diagnostic imaging (Kundel & Polansky, 2003), map comparison (Visser & Nijs, 2006) and content analysis (Krippendorff, 2004).…”

Section: Cohen's Kappamentioning

confidence: 99%

Cohen’s kappa is a weighted average

Warrens

2011

Statistical Methodology

View full text Add to dashboard Cite

Abstract:The κ coefficient is a popular descriptive statistic for summarizing an agreement table. It is sometimes desirable to combine some of the categories, for example, when categories are easily confused, and then calculate κ for the collapsed table. Since the categories of an agreement table are nominal and the order in which the categories of a table are listed is irrelevant, combining categories of an agreement table is identical to partitioning the categories in subsets.In this paper we prove that given a partition type of the categories, the overall κ-value of the original table is a weighted average of the κ-values of the collapsed tables corresponding to all partitions of that type. The weights are the denominators of the kappas of the subtables. An immediate consequence is that Cohen's κ can be interpreted as a weighted average of the κ-values of the agreement tables corresponding to all non-trivial partitions.The κ-value of the 2×2 table that is obtained by combining all categories other than the one of current interest into a single "all others" category, reflects the reliability of the individual category. Since the overall κ-value is a weighted average of these 2×2 κ-values the category reliability indicates how a category contributes to the overall κ-value. It would be good practice to report both the overall κ-value and the category reliabilities of an agreement table.

show abstract

Coefficient Kappa: Some Uses, Misuses, and Alternatives

Cited by 1,170 publications

References 13 publications

Use of simulation‐based mastery learning to improve the quality of central venous catheter placement in a medical intensive care unit

Use of simulation‐based mastery learning to improve the quality of central venous catheter placement in a medical intensive care unit

Cohen’s kappa can always be increased and decreased by combining categories

Cohen’s kappa is a weighted average

Contact Info

Product

Resources

About