The Garden of Forking Paths in Visualization: A Design Space for Reliable Exploratory Visual Analytics

Abstract

Tukey emphasized decades ago that taking exploratory findings as confirmatory is “destructively foolish”. We reframe recent conversations about the reliability of results from exploratory visual analytics—such as the multiple comparisons problem—in terms of Gelman and Loken’s garden of forking paths to lay out a design space for addressing the forking paths problem in visual analytics. This design space encompasses existing approaches to address the forking paths problem (multiple comparison correction) as well as solutions that have not been applied to exploratory visual analytics (regularization). We also discuss how perceptual bias correction techniques may be used to correct biases induced in analysts’ understanding of their data due to the forking paths problem, and outline how this problem can be cast as a threat to validity within Munzner’s Nested Model of visualization design. Finally, we suggest paper review guidelines to encourage reviewers to consider the forking paths problem when evaluating future designs of visual analytics tools.

Concept-Driven Visual Analytics

“…More generally, it seems desirable to allow users to partially and iteratively specify their model with feedback from the interface. However, if iterative model specification is enabled, the issue of multiple comparisons should be considered carefully; as multiple similar models are tested iteratively, the analyst runs a higher risk of making false discoveries [38,58]. Tools should therefore track the number of model revisions or the number of data 'peeks', thereby allowing analysts to make reasonable model adjustments while calling out the potential for false discovery and overfitting.…”

Exploring the Effects of Aggregation Choices on Untrained Visualization Users' Generalizations From Data

“…[CG14]), we believe that the ambiguity of confidence as a construct makes confidence a noisy signal at best [HQC*19]. As an alternative, visualization researchers have suggested understanding threats to the reliability of conclusions drawn during EDA more broadly as an example of Bayesian inference [HH18] or a garden of forking paths in which ‘model overfitting’ (overconfidence in trends observed in a sample) can be mitigated using regularization or visual bias corrections [PK18]. Our experiment results lead us to believe that a better approach to evaluating the integrity of EDA may be to elicit intervals for any stated effects directly from the analyst.…”

Investigating Effects of Visual Anchors on Decision‐Making about Misinformation

“…While the training process differed between groups, all users received the same interface. As argued by Pu and Kay [PK18], design may have a significant effect on the forking paths problem as well. A future study could provide control interface layouts to identify the marginal value of each view in the decision‐making process (e.g., testing whether the strategy cues with only the Tweet view – which mimics everyday social media usage – can measure a baseline accuracy).…”

“…Zgraggen et al . [ZZZK18] find too much freedom in visualization systems can lead to spurious insights and high rates of false discoveries, also known as the multiple comparisons problem or the forking paths problem [PK18]. Pu and Kay [PK18] define the forking paths problem in visualizations as “unaddressed flexibility in data analysis that leads to unreliable conclusions.” They argue cognitive biases may be one reason for users' susceptibility to the forking paths problem.…”

“…[ZZZK18] find too much freedom in visualization systems can lead to spurious insights and high rates of false discoveries, also known as the multiple comparisons problem or the forking paths problem [PK18]. Pu and Kay [PK18] define the forking paths problem in visualizations as “unaddressed flexibility in data analysis that leads to unreliable conclusions.” They argue cognitive biases may be one reason for users' susceptibility to the forking paths problem. In this paper, we consider the problem within the scope of one such cognitive bias, anchoring bias, and the possible effect pre‐task training can have on the complex decision‐making task of social media misinformation identification using a CMV VA system.…”