Reliability and Inter-rater Reliability in Qualitative Research

Abstract

What does reliability mean for building a grounded theory? What about when writing an auto-ethnography? When is it appropriate to use measures like inter-rater reliability (IRR)? Reliability is a familiar concept in traditional scientific practice, but how, and even whether to establish reliability in qualitative research is an oft-debated question. For researchers in highly interdisciplinary fields like computer-supported cooperative work (CSCW) and human-computer interaction (HCI), the question is particularly complex as collaborators bring diverse epistemologies and training to their research. In this article, we use two approaches to understand reliability in qualitative research. We first investigate and describe local norms in the CSCW and HCI literature, then we combine examples from these findings with guidelines from methods literature to help researchers answer questions like: "should I calculate IRR?" Drawing on a meta-analysis of a representative sample of CSCW and HCI papers from 2016-2018, we find that authors use a variety of approaches to communicate reliability; notably, IRR is rare, occurring in around 1/9 of qualitative papers. We reflect on current practices and propose guidelines for reporting on reliability in qualitative research using IRR as a central example of a form of agreement. The guidelines are designed to generate discussion and orient new CSCW and HCI scholars and reviewers to reliability in qualitative research.

“Disadvantaged in the American-dominated internet”: Sex, Work, and Technology

“…All interviews were double coded by two researchers who met to discuss the themes and codes after each set of two to three interviews. Because the interviewers reviewed every independently-coded transcript together, we do not present inter-rater reliability [44,45].…”

Anonymity vs. Familiarity: Self-Disclosure and Privacy in Social Virtual Reality

“…We sought first-person, subjective, narrative accounts of their experiences in the interviews and identified recurring themes. Our analytical procedures focused on eventually yielding concepts and themes (recurrent topics or meanings that represent a phenomena) rather than agreement -because even if coders agreed on codes, they may interpret the meaning of those codes differently [37]. Therefore, we did not seek inter-rater reliability in our analysis but endeavored to identify recurring themes of interest, detect relationships among them, and organize them into clusters of more complex and broader themes.…”

Emoji Accessibility for Visually Impaired People

“…4) Defining themes: The same authors reviewed the final coding to identify similarities that allowed thematic grouping. We collated codes into themes and therefore did not calculate survey inter-rater reliability because codes were not the final outcome of our analysis [43].…”