Optimal data selection: Revision, review, and reevaluation

Abstract

Since it first appeared, there has been much research and critical discussion on the theory of optimal data selection as an explanation of Wason's (1966Wason's ( , 1968 selection task (Oaksford & Chater, 1994). In this paper, this literature is reviewed, and the theory of optimal data selection is reevaluated in its light. The information gain model is first located in the current theoretical debate in the psychology of reasoning concerning dual processes in human reasoning. A model comparison exercise is then presented that compares a revised version of the model with its theoretical competitors. Tests of the novel predictions of the model are then reviewed. This section also reviews experiments claimed not to be consistent with optimal data selection. Finally, theoretical criticisms of optimal data selection are discussed. It is argued either that the revised model accounts for them or that they do not stand up under analysis. It is concluded that some version of the optimal data selection model still provides the best account of the selection task. Consequently, the conclusion of Oaksford and Chater's (1994) original rational analysis (Anderson, 1990), that people's hypothesis-testing behavior on this task is rational and well adapted to the environment, still stands.

Finding Useful Questions: On Bayesian Diagnosticity, Probability, Impact, and Information Gain.

“…Oaksford, Chater, & Grainger, 1999), in turn improving the model. Oaksford & Chater's (2003) belief model includes two hypotheses: a dependence hypothesis, that every card with an A on one side does have a 2 on the other side;…”

“…For instance, subjects may have the goal of finding out which of two hypotheses is true: h 1 , that there is a moderate correlation between X and Y, or h 0 , that X and Y are independent. Each hypothesis specifies the probability that an observation will fall in each of the cells A through D. Presence of X (glom) and Y (hulaWorn) are each rare (10% probability under both h 1 and h 0 ) in the model, corresponding to several researchers' findings that subjects usually assume rarity in related tasks (Anderson & Sheu, 1995;McKenzie & Mikkelsen, 2000;McKenzie, Ferreira, Mikkelsen, McDermott, & Skrable, 2001;Oaksford & Chater, 2003). McKenzie and Mikkelsen calculated the log 2 diagnosticity of an observation in each cell, relative to their probability model.…”

“…Several researchers explicitly make this point (Baron, 1985;Klayman & Ha, 1987;Klayman, 1987;Slowiaczek et al, 1992;Over & Jessop, 1998;Oaksford & Chater, 2003 (Skov & Sherman, 1986;Klayman & Ha, 1987, 1989Gorman, Stafford & Gorman, 1987;Devine, Hirt, & Gehrke, 1990; Finding useful questions 30 Slowiaczek et al, 1992;reviewed in Klayman, 1995). In some situations, strategies that reduce memory load (Costa-Gomes, Crawford, & Broseta, 2001) might also be used.…”

“…Tables in finding useful questions 58 Table 7 Correlation between three sampling norms and subjects' average ratings in Baron et al's (1988) Note. Each card's usefulness was calculated with respect to Oaksford and Chater's (2003) belief model, with P(dependence hypothesis)=0.5, P(error)=0.1, P(A)=0.22, and P(2)=0.27. Note.…”

Missing the dog that failed to bark in the nighttime: on the overestimation of occurrences over non-occurrences in hypothesis testing

“…The influence of the rarity of the absent clues Rarity effects concern the apportionment of increased attention to rare events in contrast to common ones (e.g., Feeney, Evans & Clibbens, 2000;Feeney, Evans & Venn, 2008;Green & Over, 2000;McKenzie & Mikkelsen, 2000Oaksford & Chater, 1994;2003;in legal contexts, for example, see Loftus, 1976;Wells & Lindsay, 1980). We included in our initial predictions a hypothesis that was based on rarity effects, conjecturing that participants would possibly pay more heed to absent clues when they were rare in comparison to present ones.…”