Confirmation, disconfirmation, and information in hypothesis testing.

Abstract

Strategies for hypothesis testing in scientific investigation and everyday reasoning have interested both psychologists and philosophers. A number of these scholars stress the importance of disconnrmation in reasoning and suggest that people are instead prone to a general deleterious "confirmation bias." In particular, it is suggested that people tend to test those cases that have the best chance of verifying current beliefs rather than those that have the best chance of falsifying them. We show, howevei; that many phenomena labeled "confirmation bias" are better understood in terms of a general positive test strategy. With this strategy, there is a tendency to test cases that are expected (or known) to have the property of interest rather than those expected (or known) to lack that property. This strategy is not equivalent to confirmation bias in the first sense; we show that the positive test strategy can be a very good heuristic for determining the truth or falsity of a hypothesis under realistic conditions. It can, however, lead to systematic errors or inefficiencies. The appropriateness of human hypothesis-testing strategies and prescriptions about optimal strategies must be understood in terms of the interaction between the strategy and the task at hand.

Finding Useful Questions: On Bayesian Diagnosticity, Probability, Impact, and Information Gain.

“…Several researchers explicitly make this point (Baron, 1985;Klayman & Ha, 1987;Klayman, 1987;Slowiaczek et al, 1992;Over & Jessop, 1998;Oaksford & Chater, 2003 (Skov & Sherman, 1986;Klayman & Ha, 1987, 1989Gorman, Stafford & Gorman, 1987;Devine, Hirt, & Gehrke, 1990; Finding useful questions 30 Slowiaczek et al, 1992;reviewed in Klayman, 1995). In some situations, strategies that reduce memory load (Costa-Gomes, Crawford, & Broseta, 2001) might also be used.…”

Diagnostic hypothesis generation and human judgment.

“…Thus, we assume that both semantic memory and episodic memory represent information corresponding to hypothesis, data, and context information. For the sake of generality, the term hypothesis (H) is used to refer to anything about which the decision maker wants to make an inference (e.g., intentions, personality traits, causes, categories, explanations, solutions, stereotypes, diagnostic labels, and hypotheses about other symptoms or treatment options; Klayman & Ha, 1987). The term data (D) is used generically to refer to any piece of information that is used as the basis of inference (e.g., behavioral patterns, symptoms, or characteristics of the stimulus).…”

Relevance theory explains the selection task

“…It may be regarded as an analogue of scientific hypothesis testing, in which subjects must appreciate the need to seek conditions which could falsify their hypothesis (cf. Popper, 1959)" (Evans, 1982, p. 157; see also, among many others, Johnson-Laird & Wason, 1970b;Klayman & Ha, 1987). The fact that some 90% of subjects fail to solve the task has been seen as evidence that most people are bad at hypothesis evaluation.…”

“…Non-demonstrative confirmation in the absence of demonstrative verification, or disconfirmation in the absence of falsification, are reasonable goals too. Inductive strategies aimed at probabilistic confirmation or disconfirmation of the rule may be quite rational and they may produce different results depending on information about the distribution of the P, not-P, Q and not-Q features in the range (see Klayman and Ha, 1987;Kirby, 1994;Oaksford and Chater, 1994).…”