An Eye-Tracking Study of Statistical Reasoning With Tree Diagrams and 2 × 2 Tables

Abstract

Changing the information format from probabilities into frequencies as well as employing appropriate visualizations such as tree diagrams or 2 × 2 tables are important tools that can facilitate people’s statistical reasoning. Previous studies have shown that despite their widespread use in statistical textbooks, both of those visualization types are only of restricted help when they are provided with probabilities, but that they can foster insight when presented with frequencies instead. In the present study, we attempt to replicate this effect and also examine, by the method of eye tracking, why probabilistic 2 × 2 tables and tree diagrams do not facilitate reasoning with regard to Bayesian inferences (i.e., determining what errors occur and whether they can be explained by scan paths), and why the same visualizations are of great help to an individual when they are combined with frequencies. All ten inferences of N = 24 participants were based solely on tree diagrams or 2 × 2 tables that presented either the famous “mammography context” or an “economics context” (without additional textual wording). We first asked participants for marginal, conjoint, and (non-inverted) conditional probabilities (or frequencies), followed by related Bayesian tasks. While solution rates were higher for natural frequency questions as compared to probability versions, eye-tracking analyses indeed yielded noticeable differences regarding eye movements between correct and incorrect solutions. For instance, heat maps (aggregated scan paths) of distinct results differed remarkably, thereby making correct and faulty strategies visible in the line of theoretical classifications. Moreover, the inherent structure of 2 × 2 tables seems to help participants avoid certain Bayesian mistakes (e.g., “Fisherian” error) while tree diagrams seem to help steer them away from others (e.g., “joint occurrence”). We will discuss resulting educational consequences at the end of the paper.

Different Visualizations Cause Different Strategies When Dealing With Bayesian Situations

“…Thus, the pre-Bayes strategy is highly dependent on the Bayesian situation and the concrete frequencies in this situation. This is apparent also in the results of Bruckmaier et al (2019), who used two situations with h 1 /(d 1 + d 3 ) > 1 and, accordingly, found nearly no pre-Bayes strategy. In our study, we used situations with h 1 /(d 1 + d 3 ) > 1, and situations with Zhu and Gigerenzer (2006), we expect few answers representing the pre-Bayes strategy in the first case.…”

“…Since the study was based on an eye-tracking method, the study included a very small sample size of 24 students. Bruckmaier et al (2019) found only strategies discussed so far for the students' Bayesian reasoning to be supported by natural frequencies and a tree diagram or 2 × 2table. The findings concerning the tree diagram supported the hypothesis that the hierarchy of the tree diagram triggers people to identify a subset-set relation in different levels of the tree.…”

“…Thus, a set inclusion following this hierarchy, such as (H ∩ D) ⊆ H, is transparent (cf. also the findings of Bruckmaier et al, 2019). Concerning the solution P (H|D) of a Bayesian reasoning task, the set H ∩ D is given by a single node, but the set D is given by two nodes representing H ∩ D andH ∩ D. Since the nodes and the related branches are parts of different paths of the tree, the set inclusion (H ∩ D) ⊆ D and (H ∩ D) ⊆ D is not transparent (Böcherer-Linder and Eichler, 2017).…”

“…A tree diagram implies a hierarchy of sets (events) that are highlighted by nodes (cf. Böcherer-Linder and Eichler, 2017 ; Bruckmaier et al, 2019 ). Thus, a set inclusion following this hierarchy, such as ( H ∩ D )⊆ H , is transparent (cf.…”

“…First, our aim was to investigate visualizations that are appropriate for training, regardless of available tools such as paper and pencil, or computers (cf. Bruckmaier et al, 2019 ). This excludes visualization representing a frequency style ( Khan et al, 2015 ) from our study.…”