Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity

Hoehler, Fred K.

doi:10.1016/s0895-4356(99)00174-2

Cited by 237 publications

(184 citation statements)

References 10 publications

Supporting

Mentioning

175

Contrasting

Unclassified

Order By: Relevance

“…We therefore calculated the proportions of positive agreement as an alternative agreement measure, which showed similar results. Future studies assessing the reliability of ADE reporting are advised to recruit a more balanced group of patients experiencing and not experiencing ADEs [51]. Based on a combined approach, that is, looking at kappa values, alternative agreement measures, and additional analysis of ADEs at patient level, we conclude that our questionnaire was not sufficiently reliable at the ADEspecific level.…”

Section: Discussionmentioning

confidence: 89%

“…For ADE reporting, however, a skewed distribution is observed where many patients report no ADEs on both measurements, which decreases the kappa values used for the reliability assessment [51,52]. Formulas to adjust for such effects have been proposed, for example, the prevalence-adjusted bias-adjusted kappa [53], but their inappropriateness has also been demonstrated [51].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Development and Initial Validation of a Patient-Reported Adverse Drug Event Questionnaire

et al. 2013

View full text Add to dashboard Cite

Background Direct patient reporting of adverse drug events (ADEs) is relevant for the evaluation of drug safety. To collect such data in clinical trials and postmarketing studies, a valid questionnaire is needed that can measure all possible ADEs experienced by patients. Objective Our aim was to develop and test a generic questionnaire to identify ADEs and quantify their nature and causality as reported by patients. Methods We created a draft list of common ADEs in layterms, which were classified in body categories and mapped to the Medical Dictionary for Regulatory Activities (MedDRA Ò ) terminology. Questions about the nature and causality were derived from existing questionnaires and causality scales. Content validity was tested through cognitive debriefing, revising the questionnaire in an iterative process. Feasibility and reliability were assessed using a Web-based version of the questionnaire. Patients received the questionnaire twice. Feasibility was assessed by the reported time needed for completion and ease of use. Reliability was calculated using Cohen's kappa and proportion of positive agreement (PPA) on: (1) any ADE at patient level; (2) similar ADEs at MedDRA Ò System Organ Class level; and (3) the same ADE at ADE-specific level.Results In the development phase, 28 patients with type 2 diabetes or asthma/chronic obstructive pulmonary disease (COPD) participated. Questions and answer options were rephrased, layout was improved, and changes were made in the classification of ADEs. The final questionnaire consisted of 252 ADEs organized in 16 body categories, and included 14 questions per reported ADE. A total of 135 patients using a median of five different drugs completed the Web-based questionnaire twice. The median completion time was 15 min for patients not reporting any ADE, and 30 min for patients reporting at least one ADE. Three quarters of the patients found the questionnaire easy to use. Test-retest reliability was acceptable at patient level (j = 0.50, PPA 0.64) and at MedDRA Ò System Organ Class level (j = 0.52, PPA 0.54), but was low at ADEspecific level (j = 0.38, PPA 0.38). Conclusion We developed a generic patient-reported ADE questionnaire and confirmed its content validity. The questionnaire was feasible and reliable for reporting any ADE and similar ADEs at MedDRA Ò System Organ Class level. Additional work is, however, needed to reliably quantify specific ADEs reported by patients.

show abstract

Section: Discussionmentioning

confidence: 89%

Section: Discussionmentioning

confidence: 99%

Development and Initial Validation of a Patient-Reported Adverse Drug Event Questionnaire

et al. 2013

View full text Add to dashboard Cite

show abstract

“…For example, in situations where physicians chose a particular readmission contributor infrequently, chance agreement is high and kappa is accordingly reduced. While this Bpenalty^is felt by many to be appropriate, [18][19][20] we attempted to minimize this phenomenon by reporting kappa values only for aggregate categories of factors predicting readmissions. The fact that kappa was generally low even for highly prevalent selections argues that the prevalence phenomenon does not explain the low rates of agreement we observed.…”

Section: Discussionmentioning

confidence: 99%

Physician Perspectives on Factors Contributing to Readmissions and Potential Prevention Strategies: A Multicenter Survey

et al. 2016

View full text Add to dashboard Cite

BACKGROUND:The transition out of the hospital is a vulnerable time for patients, relying heavily on communication and coordination of resources across care settings. Understanding the perspectives of inpatient and outpatient physicians regarding factors contributing to readmission and potential preventive strategies is crucial in designing appropriately targeted readmission prevention efforts. OBJECTIVE: To examine and compare inpatient and outpatient physician opinions regarding reasons for readmission and interventions that might have prevented readmission. DESIGN: Cross-sectional multicenter study. PARTICIPANTS: We identified patients readmitted to general medicine services within 30 days of discharge at 12 US academic medical centers, and surveyed the primary care physician (PCP), discharging physician from the index admission, and admitting physician from the readmission regarding their endorsement of pre-specified factors contributing to the readmission and potential preventive strategies. MAIN MEASURES: We calculated kappa statistics to gauge agreement between physician dyads (PCP-discharging physician, PCP-admitting physician, and admitting-discharging physician). KEY RESULTS: We evaluated 993 readmission events, which generated responses from 356 PCPs (36 % of readmissions), 675 discharging physicians (68 % of readmissions), and 737 admitting physicians (74 % of readmissions). The most commonly endorsed contributing factors by both PCPs and inpatient physicians related to patient understanding and ability to self-manage. The most commonly endorsed preventive strategies involved providing patients with enhanced post-discharge instructions and/ or support. Although PCPs and inpatient physicians endorsed contributing factors and potential preventive strategies with similar frequencies, agreement among the three physicians on the specific factors and/or strategies that applied to individual readmission events was poor (maximum kappa 0.30). CONCLUSIONS: Differing opinions among physicians on factors contributing to individual readmissions highlights the importance of communication between inpatient and outpatient providers at discharge to share their different perspectives, and suggests that multi-faceted, broadly applied interventions may be more successful than those that rely on individual providers choosing specific services based on perceived risk factors.

show abstract

“…Although the residual also comprises "conventional" error due to unsystematic impacts, we would argue that most of such unsystematic impacts may be considered to be produced by contextual particularities. An interpretation of the 23 residual predominance could be that even with "optimal" raters and items, there is a limit as to how much the level of agreement can be improved when contextual circumstances are not sufficiently controlled for. Even so, the result also indicates that the level of agreement can be substantially improved by identifying and counteracting those characteristics of the components tending to negatively influence agreement.…”

Section: Discussionmentioning

confidence: 99%

“…Barrier prevalence estimate was classified in five distributional categories, 0-20%, 21-40%, 41-60%, 61-80% and 81-100%. Moreover, following a suggestion that a balanced prevalence near 50% [23] should be targeted for the "fairest" assessment of the level of agreement, the items with prevalence estimate in the interval 41-60% were highlighted in an extended analysis. Due to the results of the predictors of agreement variation analysis described below, splitting the mean levels by rater and item characteristics did not include housing adaptation experience.…”

Section: Level Of Agreement Analysismentioning

confidence: 99%

Unfolding the phenomenon of interrater agreement: a multicomponent approach for in-depth examination was proposed

Slaug

Schilling

Helle

et al. 2012

Journal of Clinical Epidemiology

View full text Add to dashboard Cite

Objective: The overall objective was to unfold the phenomenon of inter-rater agreement: to identify potential sources of variation in agreement data and to explore how they can be statistically accounted for. The ultimate aim was to propose recommendations for in-depth examination of agreement, in order to improve the reliability of assessment instruments.Study Design and Setting: Utilizing a sample where 10 rater pairs had assessed the presence/absence of 188 environmental barriers by a systematic rating form, a raters items dataset was generated (N=1,880). In addition to common agreement indices, relative shares of agreement variation were calculated. Multilevel regression analysis was carried out, using rater and item characteristics as predictors of agreement variation.Results: Following a conceptual decomposition, the agreement variation was statistically disentangled into relative shares. The raters accounted for 6-11%, the items for 32-33% and the residual for 57-60% of the variation. Multilevel regression analysis showed barrier prevalence and raters" familiarity with using standardized instruments to have the strongest impact on agreement.Conclusion: Supported by a conceptual analysis we propose an approach of in-depth examination of agreement variation, as a strategy for increasing the level of inter-rater agreement. By identifying and limiting the most important sources of disagreement, ultimately instrument reliability can be improved.

show abstract

Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity

Cited by 237 publications

References 10 publications

Development and Initial Validation of a Patient-Reported Adverse Drug Event Questionnaire

Development and Initial Validation of a Patient-Reported Adverse Drug Event Questionnaire

Physician Perspectives on Factors Contributing to Readmissions and Potential Prevention Strategies: A Multicenter Survey

Unfolding the phenomenon of interrater agreement: a multicomponent approach for in-depth examination was proposed

Contact Info

Product

Resources

About