An Analysis of the Effect of Graph Construction Disclaimers on Visual Analysis

Effects of portable interventions on school psychologists’ graph-rating inconsistency.

“…This support was limited to reducing inconsistency in ratings of effect size magnitude and not decisions regarding whether an effect existed. Although the effect size was small, the potential promise of this approach may also be supported by recent findings that providing visual analysts with disclaimers about graph manipulations did not impact their ratings of effect size magnitude (Radley & Dart, 2023). These brief prompts may be helpful for reducing rating inconsistency in circumstances when the standardization of graph construction is inhibited.…”

“…However, the utility of this approach for reducing inconsistency in applied decision making is unclear. Practitioners often make decisions that are more similar to identifying whether the effects were meaningful or not (e.g., continue, modify, or discontinue intervention) than judgements regarding the magnitude of an effect (Radley & Dart, 2023). Judgements regarding the magnitude of treatment effects may better inform decisions regarding intervention selection and not intervention response.…”

“…For example, variations in the scaling of the Y -axis (Dart & Radley, 2017), the scaling of the X -axis (Dart & Radley, 2023), and the number of data points per X - to Y -axis ratio (DPPXYR; Radley et al, 2018), all influenced visual analysts’ identification of treatment effects and ratings regarding the magnitude of treatment effects. In fact, the manner in which graphs were assembled (e.g., Y -axis range, DPPXYR) had an impact on visual analysts’ judgements of the magnitude of effects even when the graphs were accompanied by a disclaimer that identified that graph manipulation was used to magnify intervention effects (Radley & Dart, 2023). Recent reviews by Peltier et al (2021, 2022) suggested that published SCD graphs generally had appropriate ordinate scaling, but DPPXYR values that could lead to inflated Type I error rates.…”

“…One example of a task-relevant reminder is a disclaimer that a graph has been manipulated from standardization. Radley and Dart (2023) experimentally tested such an intervention that warned participants when graph axes had been manipulated. This disclaimer failed to attenuate inflation effects of compressed graph axes.…”

The effect of the level of data presentation on visual analysts’ decisions.

“…First, researchers should clearly describe how many measurement occasions are represented within each data point in their linear graphs of single-case data. Although recent research does not support the use of disclosures or disclaimers to mitigate the effects, these kinds of graphical features have on visual analysts’ decisions (e.g., Radley & Dart, 2023), obscuring such information may make it difficult to fully contextualize a single-case data set. Second, both researchers and practitioners are encouraged to consider using the most microscopic presentation of their data that is feasible given the number of data points.…”