High agreement but low Kappa: I. the problems of two paradoxes

Feinstein, Alvan R.; Cicchetti, Domenic V.

doi:10.1016/0895-4356(90)90158-l

Cited by 2,396 publications

(1,618 citation statements)

References 9 publications

Supporting

Mentioning

1,496

Contrasting

Unclassified

Order By: Relevance

“…and "How often after hospice became involved did you feel you were on your own to advocate for the patient? "-all low kappas (≤0.40) could be explained by the skewed marginal effects, as described by Feinstein and Cicchetti (16,17), and explained above, in which a low kappa results despite high percent agreement in the data. We report the p-positive and p-negative values for items with skewed marginal effects in Table 2.…”

Section: Resultsmentioning

confidence: 74%

“…As recommended by Landis and Koch (15), we considered a kappa of less than 0.40 to indicate poor agreement. Feinstein and Cicchetti noted an important paradox of high agreement but low kappa as a result of imbalances in marginal totals (16,17). This paradox is explained by the fact that kappa statistics are dependent on the prevalence of the finding under observation.…”

Section: Analytic Approachmentioning

confidence: 99%

See 1 more Smart Citation

Timing of Survey Administration After Hospice Patient Death: Stability of Bereaved Respondents

DiBiasio

Clark

Gozalo

et al. 2015

Journal of Pain and Symptom Management

View full text Add to dashboard Cite

Context-The Centers for Medicare & Medicaid Services have elected to include a bereaved family member survey in public reporting of hospice quality data as mandated in the Affordable Care Act. However, it is not known what timepoint after death offers the most reliable responses.Objectives-To examine the stability of bereaved family members' survey responses when administered three, six and nine months after hospice patient death.Methods-Bereaved family members from six geographically diverse hospices were interviewed three, six, and nine months after patient death. All respondents completed a core survey. Those whose family member died at home, in a free-standing inpatient unit, or in a nursing home also completed a site-specific module. Stability was based on top-box scoring of each item with kappa statistics, and multivariable regression models were used to assess directionality and predictors of change. To analyze the effects of grief, we assessed response stability among respondents at least one standard deviation from the mean change in grief between three and six months.Results-We had 1532 surveys (536 three-month surveys, 529 six-month surveys, and 467 ninemonth surveys) returned by 643 respondents (average age 61.7 years, 17.4% Black, 50.5% a child respondent) about hospice decedents (55.3% female, average age 78.6 years, 57.0% non-cancer, 40.0% at home.) The average kappa for core items between three and nine months was 0.54 (range: 0.42-0.74), 0.58 (0.41-0.69) for home-specific items, and 0.54 (0.39-0.63) for nursing home. Even among individuals demonstrating large grief changes, core items demonstrated moderate to high stability over time.Conclusion-Bereaved family member responses are stable between three and nine months after the death of the patient.

show abstract

Section: Resultsmentioning

confidence: 74%

Section: Analytic Approachmentioning

confidence: 99%

Timing of Survey Administration After Hospice Patient Death: Stability of Bereaved Respondents

DiBiasio

Clark

Gozalo

et al. 2015

Journal of Pain and Symptom Management

View full text Add to dashboard Cite

show abstract

“…It is known that when the table's marginal totals are substantially unbalanced, kappa is low in spite of high agreement between raters. 24 The recommendation has been made to either report proportionate agreement with respect to a positive (`Yes') decision and with respect to a negative (`No') decision, 25 or kappa and percentage agreement. 26 As is clear from Table 4, except for the decision whether the paper deals with human subjects, the per cent agreement was also low; therefore, kappa was used in Table 5.…”

Section: Discussionmentioning

confidence: 99%

Searching the literature for information on traumatic spinal cord injury: the usefulness of abstracts

Dijkers

2003

Spinal Cord

View full text Add to dashboard Cite

Study design: Systematic review of abstracts of published papers presumed to contain information on chronic pain in persons with spinal cord injury (SCI). Objectives: To determine to what degree papers on SCI are abstracted in such a way that they can be retrieved, and evaluated as to the paper's applicability to a reader's questions. Setting: US -academic department of rehabilitation medicine. Methods: 868 abstracts published in Medline were independently examined by two out of 13 screeners, who answered four questions on the subjects and nature of the paper with`Yes', No' or`insu cient information'. Frequency of ratings`insu cient information', and screener agreement were evaluated as a ected by screener and abstract/paper characteristics. Results: Screeners could not determine whether the paper dealt with persons with traumatic SCI for 37% of abstracts; whether chronic pain was a topic could not be determined in 18%. Physicians were less willing than other disciplines to assign`insu cient information'. Screener agreement was better than chance, but not at the level suggested for quality measurement. Screener discipline and task experience did not make a di erence, nor did abstract length, structure, or decade of publication of the paper. Conclusion: Authors need to improve the quality of abstracts to make retrieval and screening of relevant papers more e ective and e cient.

show abstract

“…The proportion of observed agreement (p O ) and Cohen's Kappa (j) were calculated as measures of concurrent validity. Since p O and j show no insight into the agreement between the positive and negative answers and because the j statistic is considered unstable as it is strongly influenced by the observed proportions of individuals who fall in each category of classification (Speklé et al 2009;Perreault et al 2008;Juul-Kristensen et al 2006;Salerno et al 2000;Feinstein and Cicchetti 1990), p positive (p pos ) and p negative (p neg ) were also calculated as extra means of assessing the agreement (Speklé et al 2009;Feinstein and Cicchetti 1990;Cicchetti and Feinstein 1990). According to Cicchetti and Feinstein (Cicchetti and Feinstein 1990), the observed proportion of positive agreement (p pos ) can be calculated as the ratio of the actual number of subjects that the questionnaire and the occupational physician agree on having symptoms over the average number of subjects with symptoms that were identified by the questionnaire and the occupational physician ((cases questionnaire ?…”

Section: Statistical Analysesmentioning

confidence: 99%

Concurrent validity of questions on arm, shoulder and neck symptoms of the RSI QuickScan

Hoozemans

Speklé

Dieën

2012

Int Arch Occup Environ Health

View full text Add to dashboard Cite

Purpose The objective was to determine the concurrent validity of questions on arm, shoulder and neck symptoms of an Internet-based questionnaire. In addition, the interobserver reliability of physical examinations by occupational physicians was investigated. Methods A total of 160 employees of a Dutch occupational health service were approached, of which 106 participated. Right after the assessment of arm, shoulder and neck symptoms using a self-administered questionnaire, each participant was examined by two occupational physicians. The presence of symptoms in the past 7 days was compared to the physical examinations. The participation of two occupational physicians allowed us to study also the inter-observer reliability. Results Overall, the concurrent validity of the symptom questions of the questionnaire can be defined as poor to moderate with j values between 0.16 and 0.53. Detecting the presence of symptoms (p pos ) could be considered as moderately valid with values below 0.60, but the p neg shows that the concurrent validity for detecting the absence of arm, shoulder or neck symptoms can be considered sufficient with values above 0.69. The agreement between occupational physicians can, with a few exceptions, be considered as moderate with j values below 0.60. The agreement was sufficient for detecting the absence of symptoms (p neg [ 0.7). Conclusions The agreement between the symptom questions of the questionnaire and physical examinations of occupational physicians can be considered as poor to moderate. The results are comparable to what is generally reported in the literature. Future studies should be aimed at gaining more fundamental knowledge about the possible conceptual differences between self-reported symptoms and symptoms assessed using physical examinations. Moreover, it is advisable to improve the inter-observer reliability of physical examinations as applied in the present study.

show abstract

High agreement but low Kappa: I. the problems of two paradoxes

Cited by 2,396 publications

References 9 publications

Timing of Survey Administration After Hospice Patient Death: Stability of Bereaved Respondents

Timing of Survey Administration After Hospice Patient Death: Stability of Bereaved Respondents

Searching the literature for information on traumatic spinal cord injury: the usefulness of abstracts

Concurrent validity of questions on arm, shoulder and neck symptoms of the RSI QuickScan

Contact Info

Product

Resources

About