Visual and Statistical Methods to Calculate Intercoder Reliability for Time-Resolved Observational Research

Abstract

While calculating intercoder reliability (ICR) is straightforward for text-based data, such as for interview transcript excerpts, determining ICR for naturalistic observational video data is much more complex. To date, there have been few methods proposed in literature that are robust enough to handle complexities such as the occurrence of simultaneous event complexity and partial agreement by the raters. This is especially important with the emergence of high-resolution video data, which collects nearly continuous or continuous observational data in naturalistic settings. In this paper, we present three approaches to calculating ICR. First, we present the technical approach to clean and compare two coders’ results such that traditional metrics of ICR (e.g., Cohen’s κ, Krippendorff’s α, Scott’s Π) can be calculated, methods previously unarticulated in literature. However, these calculations are intensive, requiring significant data manipulation. As an alternative, this paper also proposes two novel methods to calculate ICR by algorithmically comparing visual representations of each coders’ results. To demonstrate efficacy of the approaches, we employ all three methods on data from two separate ongoing research contexts using observational data. We find that the visual methods perform as well as the traditional measures of ICR and offer significant reduction in the work required to calculate ICR, with an added advantage of allowing the researcher to set thresholds for acceptable agreement in lag time. These methods may transform the consideration of ICR in other studies across disciplines that employ observational data.

A Day in the Life: Characterization of Doctoral Bench Research in Synthetic Chemistry Using Phenomenological Case Studies

“…Unfortunately, calculating inter-rater and intra-rater reliability (or, more properly termed for this type of nominal data, intercoder/intracoder reliability) is highly challenging for complex naturalistic observational video data featuring overlapping open codes and an evolving, living codebook. Attempts to quantify such comparisons present problematic implications and oversimplification due both to the nature of grounded theory research and the statistical issues arising from a large, complex codebook with multiple rare codes (e.g., prevalence discrepancies). ,− However, visual analysis of the coding (see SI) illustrated an apparently high degree of agreement between the participants and author in assigning MA codes, especially after accounting for restrictions that the author operated under in assigning codes which the participants did not (Box ). The participants sometimes provided more MA codes during their verbal open coding because they could recall cognitive decisions they were making or report on what their eyes were doing behind the camera.…”