Misjudgment of interrupted time-series graphs due to serial dependence: Replication of

Abstract

Interrupted time-series graphs are often judged by eye. Such a graph might show, for example, patient symptom severity (y) on each of several days (x) before and after a treatment was implemented (interruption). Such graphs might be prone to systematic misjudgment because of serial dependence, where random error at each timepoint persists into later timepoints. An earlier study (Matyas & Greenwood, 1990) showed evidence of systematic misjudgment, but that study has often been discounted due to methodological concerns. We address these concerns and others in two experiments. In both experiments, serial dependence increased mistaken judgments that the interrupting event led to a change in the outcome, though the pattern of results was less extreme than in previous work. Receiver operating characteristics suggested that serial dependence both decreased discriminability and increased the bias to decide that the interrupting event led to a change. This serial dependence effect appeared despite financial incentives for accuracy, despite feedback training, and even in participants who had graduate training relevant to the task. Serial dependence could cause random error to be misattributed to real change, thereby leading to judgments that interventions are effective even when they are not.

Type I error rates and power of two randomization test procedures for the changing criterion design

“…In light of the presence of autocorrelation in SCED data (Barnard-Brak et al, 2021 ; Shadish & Sullivan, 2011 ; Sideridis & Greenwood, 1997 ; Solomon, 2014 ) and its prominence in SCED simulation studies on different data analytical procedures (e.g., Baek & Ferron, 2013 ; Bishara et al, 2021 ; De & Onghena, 2022 ; Hedges et al, 2023 ; Petit-Bois et al, 2016 ; Smith et al, 2012 ), including randomization tests (Bouwmeester & Jongerling, 2020 ; Ferron & Onghena, 1996 ; Ferron & Sentovich, 2002 ; Levin et al, 2012 ; Manolov, 2019 ), it was decided to segment the results per level of autocorrelation in Figs. 8 , 9 , 10 , 11 , 12 , and 13 .…”

A Proposal for the Assessment of Replication of Effects in Single-Case Experimental Designs

“…Second, in terms of the advantages of the modified Brinley plot in relation to the common time-series line graphs, several aspects need to be addressed. First, the agreement between visual analysts inspecting time-series plots has been found to be insufficient (see Ninci et al, 2015, for a meta-analysis, and also Bishara et al, 2021;Tarlow et al, 2021). Second, the modified Brinley plot is not affected by graphical features such as the ratio between x-axis and the y-axis (x:y ratio; Kubina et al, 2017), given that it is square by definition.…”

“…That is, if the study is replicated and a positive effect of the intervention is observed repeatedly, the degree of uncertainty will be reduced. Replication is not only relevant in applied domains (e.g., replication of intervention effects), but also when carrying out methodological studies on SCED data analytical procedures (e.g., Bishara et al, 2021;Falligant et al, 2020). Thus, it is important to have an objective way of defining whether the results of different replications agree or not.…”