The kappa paradox

Bexkens, Rens; Claessen, Femke M.A.P.; Kodde, Izaäk F.; Oh, Luke S.; Eygendaal, Denise; Bekerom, Michel PJ van den

doi:10.1177/1758573218791813

Cited by 5 publications

(4 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…On the other hand, there are several authors that defend that Kappa is a useful measure of agreement, when its limitations are taken into account. For example, in [32] the authors defend the use of Kappa in a previous study, and warn that it is a useful measure if marginal distributions are considered. A similar conclusion was reached in [33], where it is said that although Kappa is not suitable in certain circumstances, it is better than the raw proportion.…”

Section: Introductionmentioning

confidence: 99%

Why Cohen’s Kappa should be avoided as performance measure in classification

2019

View full text Add to dashboard Cite

We show that Cohen’s Kappa and Matthews Correlation Coefficient (MCC), both extended and contrasted measures of performance in multi-class classification, are correlated in most situations, albeit can differ in others. Indeed, although in the symmetric case both match, we consider different unbalanced situations in which Kappa exhibits an undesired behaviour, i.e. a worse classifier gets higher Kappa score, differing qualitatively from that of MCC. The debate about the incoherence in the behaviour of Kappa revolves around the convenience, or not, of using a relative metric, which makes the interpretation of its values difficult. We extend these concerns by showing that its pitfalls can go even further. Through experimentation, we present a novel approach to this topic. We carry on a comprehensive study that identifies an scenario in which the contradictory behaviour among MCC and Kappa emerges. Specifically, we find out that when there is a decrease to zero of the entropy of the elements out of the diagonal of the confusion matrix associated to a classifier, the discrepancy between Kappa and MCC rise, pointing to an anomalous performance of the former. We believe that this finding disables Kappa to be used in general as a performance measure to compare classifiers.

show abstract

Section: Introductionmentioning

confidence: 99%

Why Cohen’s Kappa should be avoided as performance measure in classification

2019

View full text Add to dashboard Cite

show abstract

“…To address some of these issues, we have seen the rise of Cohen's Kappa and Matthews Correlation Coefficient (MCC), which tries to address a larger scope of variables to gauge classifier performance. Although the use of K has been an area of debate inside ML domains (Delgado & Tibau, 2019) and outside them (Bexkens et al, 2018), we see a much more positive reception for MCC (Chicco & Jurman, 2020).…”

Section: The Limitations Of ML Metrics In Financial Crime Analyticsmentioning

confidence: 79%

An approach to benchmark fraud detection algorithms in the COVID-19 era

Figueroa¹,

Turner-Szymkiewicz²,

Cárdenas-Rodriguez³

et al. 2021

RLESD

View full text Add to dashboard Cite

To address the challenges in the fight against financial crime, particularly in the COVID-19 pandemic context, this paper focuses on financial synthetic data and the use of a reliable benchmark tool to test fraud detection algorithms. Compliance departments at financial institutions face the challenge of reducing the number of innocent people erroneously accused of fraud. To cope with this problem financial institutions are exploring the application of machine learning fraud detection algorithms and data analysis technologies to develop a more accurate and precise fraud detection system. However, approaches to streamlining and automating banks’ monitoring and testing processes is challenging as there is no consensus on a benchmark. We explore the relevance of measuring the applicability of a financial crime benchmark in the presence of a growing digital financial sector, such as in the case of Mexico. This study is particularly important due to serious threats that are faced by a rapidly developing financial system (2019 Mexican Central Bank Report). These risks have been further exacerbated as a result of the COVID-19 pandemic accelerating the shift towards digital payments.

show abstract

“…While these observations seem counter-intuitive, the phenomenon has been explained as a kappa paradox which results from imbalanced marginal distributions of the data [25,26]. However, as explained by Bexkens et al [27], this is not a limitation of the kappa coefficient but rather a logical consequence of its purpose to correctly interpret agreement adjusted for agreement by chance alone. We therefore opted to report both concordances and kappa statistics, while at the same time showing all underlying numbers to allow for detailed insights in data.…”

Section: Discussionmentioning

confidence: 99%

Combining data to perform population-based observational studies: know your sources. The case of thyroid cancer in Belgium

Decallonne

Snyers²,

Elaut

et al. 2022

Arch Public Health

View full text Add to dashboard Cite

Background Large scale observational studies are crucial to study thyroid cancer incidence and management, known to vary in time and place. Combining cancer registry data with other data sources enables execution of population-based studies, provided data sources are accurate. The objective was to compare thyroid tumour and treatment information between the available data sources in Belgium. Methods We performed a retrospective national population-based cohort study. All patients with thyroid cancer diagnosis in Belgium between 2009 and 2011 (N = 2659 patients) were retrieved from the Belgian Cancer Registry database, containing standard patient and tumour characteristics. Additionally, information was obtained from the following sources: a) detailed pathology reports b) the health insurance company database for reimbursed performed therapeutic acts (both available for N = 2400 patients) c) registration forms for performed and/or planned treatments at the time of the multidisciplinary team meeting (available for N = 1819 patients). More precisely, information was retrieved regarding characteristics of the tumour (histologic subtype, tumour size, lymph node status (source a)) and the treatment (thyroid surgery (a,b,c), lymph node dissection (a,b), postoperative administration of radioactive iodine (b,c)). Results High concordance in histological cancer subtype (> 90%), tumour size (96.2%) and lymph node involvement (89.2%) categories was found between the cancer registry database and the pathology reports. Tumour subcategories (such as microcarcinoma, tumor ≤1 cm diameter) were more specified in the pathology reports. The therapeutic act of thyroid surgery as mentioned in the pathology reports and health insurance company database was concordant in 92.7%, while reports from multidisciplinary team meetings showed 88.5% of concordance with pathology reports and 86.1% with health insurance data. With regard to postoperative radioiodine administration, reports from multidisciplinary teams and health insurance data were concordant in 76.8%. Conclusion Combining registered and/or administrative data results in sufficiently accurate information to perform large scale observational studies on thyroid cancer in Belgium. However, thorough and continuous quality control and insight in strengths and limitations of each cancer data source is crucial.

show abstract

The kappa paradox

Cited by 5 publications

References 7 publications

Why Cohen’s Kappa should be avoided as performance measure in classification

Why Cohen’s Kappa should be avoided as performance measure in classification

An approach to benchmark fraud detection algorithms in the COVID-19 era

Combining data to perform population-based observational studies: know your sources. The case of thyroid cancer in Belgium

Contact Info

Product

Resources

About