High agreement but low kappa: II. Resolving the paradoxes

Cicchetti, Domenic V.; Feinstein, Alvan R.

doi:10.1016/0895-4356(90)90159-m

Cited by 1,511 publications

(986 citation statements)

References 12 publications

Supporting

Mentioning

935

Contrasting

Unclassified

Order By: Relevance

“…and "How often after hospice became involved did you feel you were on your own to advocate for the patient? "-all low kappas (≤0.40) could be explained by the skewed marginal effects, as described by Feinstein and Cicchetti (16,17), and explained above, in which a low kappa results despite high percent agreement in the data. We report the p-positive and p-negative values for items with skewed marginal effects in Table 2.…”

Section: Resultsmentioning

confidence: 73%

“…As recommended by Landis and Koch (15), we considered a kappa of less than 0.40 to indicate poor agreement. Feinstein and Cicchetti noted an important paradox of high agreement but low kappa as a result of imbalances in marginal totals (16,17). This paradox is explained by the fact that kappa statistics are dependent on the prevalence of the finding under observation.…”

Section: Analytic Approachmentioning

confidence: 99%

“…A large p-positive and a small p-negative, or vice versa, indicates a skewed marginal distribution that is the likely cause of a low kappa in such circumstances. When we encountered low kappa values, we utilized the approach proposed by Cicchetti and Feinstein of examining the separate omnibus index for the proportion of positive and negative agreement in these instances (16).…”

Section: Analytic Approachmentioning

confidence: 99%

See 2 more Smart Citations

Timing of Survey Administration After Hospice Patient Death: Stability of Bereaved Respondents

DiBiasio

Clark

Gozalo

et al. 2015

Journal of Pain and Symptom Management

View full text Add to dashboard Cite

Context-The Centers for Medicare & Medicaid Services have elected to include a bereaved family member survey in public reporting of hospice quality data as mandated in the Affordable Care Act. However, it is not known what timepoint after death offers the most reliable responses.Objectives-To examine the stability of bereaved family members' survey responses when administered three, six and nine months after hospice patient death.Methods-Bereaved family members from six geographically diverse hospices were interviewed three, six, and nine months after patient death. All respondents completed a core survey. Those whose family member died at home, in a free-standing inpatient unit, or in a nursing home also completed a site-specific module. Stability was based on top-box scoring of each item with kappa statistics, and multivariable regression models were used to assess directionality and predictors of change. To analyze the effects of grief, we assessed response stability among respondents at least one standard deviation from the mean change in grief between three and six months.Results-We had 1532 surveys (536 three-month surveys, 529 six-month surveys, and 467 ninemonth surveys) returned by 643 respondents (average age 61.7 years, 17.4% Black, 50.5% a child respondent) about hospice decedents (55.3% female, average age 78.6 years, 57.0% non-cancer, 40.0% at home.) The average kappa for core items between three and nine months was 0.54 (range: 0.42-0.74), 0.58 (0.41-0.69) for home-specific items, and 0.54 (0.39-0.63) for nursing home. Even among individuals demonstrating large grief changes, core items demonstrated moderate to high stability over time.Conclusion-Bereaved family member responses are stable between three and nine months after the death of the patient.

show abstract

Section: Resultsmentioning

confidence: 73%

Section: Analytic Approachmentioning

confidence: 99%

Section: Analytic Approachmentioning

confidence: 99%

See 1 more Smart Citation

Timing of Survey Administration After Hospice Patient Death: Stability of Bereaved Respondents

DiBiasio

Clark

Gozalo

et al. 2015

Journal of Pain and Symptom Management

View full text Add to dashboard Cite

show abstract

“…Since k is influenced by prevalence of the characteristic being measured, agreement was measured by category-specific k and percent agreements to accommodate uncommon or low prevalent features. 5 In general, k statistics less than 0.4 are associated with relatively poor agreement, values of 0.4-0.6 moderate agreement, values of 0.6-0.8 substantial (good) agreement and values greater than 0.8 are associated with excellent (almost perfect) agreement. 6 In the second approach, the reproducibility (or accuracy) of the study pathologists' diagnoses was assessed relative to the reference standard.…”

Section: Methodsmentioning

confidence: 94%

Interobserver agreement and reproducibility in classification of invasive breast carcinoma: an NCI breast cancer family registry study

et al. 2006

View full text Add to dashboard Cite

Registry of this type; over 37 724 individuals have been enrolled to date. One activity of this Registry is the semicentralized pathologic review of tumors from all probands. Given the semicentralized nature of the review, this study was undertaken to determine the reproducibility, source(s) of classification discrepancies and stratagems to circumvent discrepancies for histologic subtyping and grading of invasive breast cancer among the reviewing pathologists. A total of 13 pathologists reviewed 35 invasive breast cancers and classified them by primary and secondary histologic type, Nottingham grade and score. Lymph-vascular space invasion, circumscribed margins, syncytial growth and lymphocytic infiltrate were also evaluated. A training session using a separate set of slides was conducted prior to the study. General agreement, in terms of categoryspecific j's and percent agreement, and accuracy of classification relative to a reference standard were determined. Classification of histologic subtype was most consistent (and accurate) for mucinous carcinoma (j ¼ 1.0), followed by tubular (j ¼ 0.8) and lobular subtypes (j ¼ 0.8). Classification of medullary subtype was moderate (j ¼ 0.4), but additional evaluation of degree of lymphocytic infiltrate, syncytial growth and circumscribed margins identified most cases. Category-specific j's were moderate to good for Nottingham grade (j ¼ 0.5-0.7), with the greatest agreement obtained in categorizing grade I (j ¼ 0.7), and grade III tumors (j ¼ 0.7). A flexible classification strategy that employs individual and combined criteria provides good interobserver agreement for invasive breast cancers with uniform, unambiguous histology and compensates for classification discrepancies in the more histologically ambiguous or heterogeneous cancers. Keywords: interobserver reproducibility; invasive breast cancer; familial breast cancer; breast/ovarian cancer family registryThe reproducibility of the classification and grading of invasive breast cancer and the cause(s) of interobserver disagreement among pathologists have not been adequately evaluated. Prior studies evaluating interobserver concordance in categorizing breast lesions have documented improved diagnostic agreement when the pathologists involved used agreed-upon criteria, 1 but other potential sources of poor interobserver agreement, such as the difficulties in the application of the individual histologic criteria, the individual pathologist's variation in use of these criteria, and most importantly, the ambiguous or borderline and heterogeneous nature of the

show abstract

“…Through this relationship, kappa coefficients depend on the prevalence of the trait under study, which limits the possibility to compare them among studies with different prevalence. Several authors (Thompson and Walter, 1988; Feinstein and Cicchetti, 1990; Cicchetti and Feinstein, 1990; Byrt et al ., 1993; de Vet et al ., 2006) proposed the use of absolute agreement measures (e.g. the proportion of items classified in the same category by the two observers) to avoid that dependency.…”

Section: Introductionmentioning

confidence: 99%

Comparing dependent kappa coefficients obtained on multilevel data

Vanbelle

2017

Biometrical J

View full text Add to dashboard Cite

Reliability and agreement are two notions of paramount importance in medical and behavioral sciences. They provide information about the quality of the measurements. When the scale is categorical, reliability and agreement can be quantified through different kappa coefficients. The present paper provides two simple alternatives to more advanced modeling techniques, which are not always adequate in case of a very limited number of subjects, when comparing several dependent kappa coefficients obtained on multilevel data. This situation frequently arises in medical sciences, where multilevel data are common. Dependent kappa coefficients can result from the assessment of the same individuals at various occasions or when each member of a group is compared to an expert, for example. The method is based on simple matrix calculations and is available in the R package “multiagree”. Moreover, the statistical properties of the proposed method are studied using simulations. Although this paper focuses on kappa coefficients, the method easily extends to other statistical measures.

show abstract

High agreement but low kappa: II. Resolving the paradoxes

Cited by 1,511 publications

References 12 publications

Timing of Survey Administration After Hospice Patient Death: Stability of Bereaved Respondents

Timing of Survey Administration After Hospice Patient Death: Stability of Bereaved Respondents

Interobserver agreement and reproducibility in classification of invasive breast carcinoma: an NCI breast cancer family registry study

Comparing dependent kappa coefficients obtained on multilevel data

Contact Info

Product

Resources

About