Kappa Agreement of Emergency Department Triage Scales; A Systematic Review and Meta-Analysis

Abstract

The aim of this review was to compare the reliability of existing triage scales. The research question was that, using kappa value as reliability measure, in adult ED patients to what extent did clinicians' triage ratings agree with eachother or with a gold standard? Methods: The search was limited to studies on adult patients (≥15 years) visiting EDs for somatic reasons and to English and Persian studies that have reported kappa. Studies solely on geriatric patients, computerized triage, telephone triage, and specialty triages were excluded. PubMed, Scopus, ISI web of knowledge, Mosby's Nursing Index, Magiran and SID, were systematically searched through August 2013. 13 articles for kappa (n=38829), 11 articles for linear weighted kappa (LWK) (n= 22697) and 12 articles for quadratic weighted kappa (QWK) (n=10915) met our inclusion criteria. Heterogeneity among studies and publication bias were checked. All pooled analyses of kappa scores for each triage scale were calculated based on random-effects model and then the scales were compared. Results: Findings suggest that five-level scales are more reliable in triaging patients in the emergency department than others (pooled kappa: 0.53, 95% CI (0.48, 0.57), LWK= 0.76, 95% CI (0.72, 0.81) and QWK = 0.74, 95% CI (0.69, 0.79)). The Canadian Triage and Acuity Scale (CTAS) and the Emergency Severity Index (ESI) have been studied more than the other scales. Conclusion: It seems necessary to set a consensus method to assess and compare the reliability of triage scales. This review suggests more specific studies on reliability of triage scales, especially on intra-rater agreement.

Emergency Department Triage in Sweden: Occurrence, Validity, Reliability and Registered Nurses' Experiences

“…Triage reliability studies at the time were commonly analysed with kappa (κ) statistics, un-weighted (κ) or weighted (κW), linear (LκW) or quadratic (QκW), indicating a need for consensus on the method, in order to assess and compare the reliability of the triage scales and systems (159), even if it there has been recommendations to apply both un-weighted and quadratic weighted kappa statistics to enable comparison (160). Weighted kappa statistics imply that different weights are given to the disagreements, which is why the weighted kappa statistics usually have higher agreement because disagreements are more likely to be only one level, not several levels (132).…”

“…As the focus of the comparative aim of the study and earlier studies of RETTS© have applied Fleiss kappa (κ) statistics (145,154), this was the natural choice. However, as there is no consensus on how to measure reliability, it is important to have provided a thorough description of the case mix and method (159,160). All analysis in the study was conducted with de-identified data, de-identified by the researcher.…”

“…As there is a lack of consensus on how to measure or study the outcome of any triage scale or system, no real evaluation or comparison can be performed (135,159). Perhaps it is time to adopt the suggestion presented by Fitzgerald et al (2010) to commit to an international triage scale that is universally understood and possible to improve through collaborative research (2).…”