Process guidelines for establishing Intercoder Reliability in qualitative studies

Abstract

Qualitative interviews are increasingly being utilized within the context of intervention trials. While there is emerging assistance for conducting and reporting qualitative analysis, there are limited practical resources available for researchers engaging in a group coding process and interested in ensuring adequate Intercoder Reliability (ICR); the amount of agreement between two or more coders for the codes applied to qualitative text. Assessing the reliability of the coding helps establish the credibility of qualitative findings. We discuss our experience calculating ICR in the context of a behavioural HIV prevention trial for young women in South Africa which involves multiple rounds of longitudinal qualitative data collection. We document the steps that we took to improve ICR in this study, the challenges to improving ICR, and the value of the process to qualitative data analysis. As a result, we provide guidelines for other researchers to consider as they embark on large qualitative projects.

How do consumers categorize new hybrid products?

“…To test intercoder reliability, a single judge, a doctoral student in management science, lent herself to the exercise of a partial coding of the data on 11% of the corpus, a rate considered sufficient for a double coding procedure by Roussel and Wacheux (, p. 105). A double coding rate between 10% and 25% of the total corpus size was used also in MacPhail, Khoza, Abler, and Ranganathan (). We defined and communicated a protocol for each descriptor, that is to say, a method that allows recognizing the descriptor in the text (Table ).…”

Evolución de la Parentalidad Positiva: Estudio longitudinal de los efectos de la aplicación de un programa de educación parental

“…Para ello se empleó un doble proceso: por un lado, se construyeron las categorías siguiendo los factores empleados en el análisis cuantitativo; por otro, se consideró seguir "códigos abiertos" o codificación "in vivo" (Kuckartz, 2014), donde a partir de la voz de los participantes se construyen los códigos, de tal manera que se puede acceder a la percepción que tenían los padres y madres sobre lo que les ha aportado el programa, y no ceñir el análisis, únicamente, a las categorías construidas a priori por el análisis cuantitativo. El proceso de codificación se realizó por parte de dos investigadores del equipo, el índice Kappa de acuerdo inter-evaluadores fue 0,62, IC al 95% (0,54, 0,69), mostrando un nivel bueno de acuerdo en el proceso de codificación (MacPhail, Khoza, Abler, & Ranganathan, 2015).…”

Ease on Down the Code

“…Krippendorff states that a minimum of three coders is required for any collaborative qualitative coding reliability and to achieve "confidence in the data beyond the measured reliability" [7]. However, as we discuss in detail in Section 2, collaboration among multiple coders is often challenging, because (1) coordinating and organizing multiple coders and several rounds of coding is complex [8], (2) achieving an acceptable ICR can be complicated [7][8][9] because ICR is impacted by factors such as number of coders, coders' skills, data size, as well as whether coders have the same understanding and interpretation of data, categories, or instructions [10], (3) when ICR is low, finding the reasons and addressing the problems is often arduous, because it requires backtracking from aggregated codes to find the individual codes and then discussing the related quotes with the coders, and (4) coders can agree with each other but still be wrong [1 and 12].…”