The influence of data characteristics on interrater agreement among visual analysts

Abstract

Visual analysis is the primary method of analyzing single‐case research data, yet relatively little is known about the variables that influence raters' decisions and rater agreement. Previous research has suggested that trend, variability, and autocorrelation may negatively affect interrater agreement, but studies have been limited by small numbers of graphs and participants whose knowledge of single‐case research was not described. The purpose of this study was to examine the main and interaction effects of two values of each of six data characteristics (e.g., level, trend, and number of data points) on agreement among visual analysts. Using data from Lanovaz and Hranchuk (2021), we examined odds ratios to identify data characteristics that influence interrater agreement. Results suggest that trend and effect size, and to a lesser extent variability, have the largest effects on interrater agreement. We discuss the implications of our results for future research on improving interrater agreement among visual analysts.

Evaluating the correspondence between expert visual analysis and quantitative methods.

“…This finding is congruent with the outcomes of Dart and Radley (2017), which showed that errors of commission were least likely when null effects were displayed. Similarly, these results mirror those of Wolfe and Seaman (2023), who found that graphs demonstrating an effect size of 0 or a large effect resulted in higher levels of interrater agreement. This is also consistent with the findings of Ninci et al (2015), who found that graphs showing small or moderate effects had lower levels of interrater agreement than those with larger or no effects.…”

“…Differences between effect size calculations and ratings of effects through visual analysis may arise due to considerations of changes in trend and variability between phases or overlap between phases, which experts strongly agreed were influential in their analysis of a single-case design graph. Wolfe and Seaman (2023) found that trend, effect size, and variability were influential in ratings of AB graphs; it is possible that these characteristics impacted raters’ judgments of graphs inconsistently in the present study. Furthermore, there is lack of consensus on the number of main effects needed to determine a functional relation—some, but not all, indicate that there should be three effects demonstrated (Wolfe & Seaman, 2023).…”

“…Wolfe and Seaman (2023) found that trend, effect size, and variability were influential in ratings of AB graphs; it is possible that these characteristics impacted raters’ judgments of graphs inconsistently in the present study. Furthermore, there is lack of consensus on the number of main effects needed to determine a functional relation—some, but not all, indicate that there should be three effects demonstrated (Wolfe & Seaman, 2023). In a recent survey, Wolfe and McCammon (2022) found differences across instructors in the number of effects needed to determine a functional relation.…”

“…Individuals using visual analysis can be prone to making errors regarding the presence of an effect (i.e., Type I errors; e.g., Matyas & Greenwood, 1990) and to over-or underestimate the size of an effect presented in graphical data (e.g., Ximenes et al, 2009). Wolfe and Seaman (2023) found that visual analysts were most influenced by trend, effect size, and variability in their judgments of AB graphs, but were only able to correctly predict agreement amongst raters 61% of the time, demonstrating that raters may respond differently to the same graph characteristics. Dart and Radley (2017) found that interpretation errors may be particularly likely when the y-axis of the graphed data is manipulated in such a way that the full scale of possible dependent variable values are not represented; this holds true even when the individuals reviewing the graphs are experts in visual analysis.…”