Intercoder Agreement in Analysis of Responses to Open-Ended Interview Questions: Examples from Tuberculosis Research

Trading sex: Voluntary or coerced? The experiences of homeless youth

“…Once these initial codebooks were developed by each rater, they were compared and standardized. This technique is consistent with methodology proposed by Carey, Morgan, and Oxtoby (1996). Each rater then independently placed the 13 respondents within these three main categories and results were again compared; interrater reliability was found to be 100%.…”

Characterising the nature of primary care patient safety incident reports in the England and Wales National Reporting and Learning System: a mixed-methods agenda-setting study for general practice

“…77 To ensure validity and reliability of coding throughout the study, regular intercoder reliability checks were undertaken on a 20% random sample of each reviewer's coding quota for every 250 reports coded. 78 Kappa statistics were calculated for each principal incident type, defined as the incident that occurred just before the harm or potential harm. A kappa of > 0.7 was sought and is consistent with previous studies of a similar nature.…”

Reliability of assessment of adherence to an antimicrobial treatment guideline

“…28 In some studies, disagreement between observers was resolved by having a single observer, which ensured consistent analysis but carried the risk of consistent wrong assessments being made. 10,14,31 Another approach described was co-review of each case, which was feasible in experimental situations but might be excessively labour intensive in continuously running programmes.…”