Information selection and use in hypothesis testing: What is a good question, and what is a good answer?

Abstract

The process of hypothesis testing entails both information selection (asking questions) and information use (drawing inferences from the answers to those questions). We demonstrate that although subjects may be sensitive to diagnosticity in choosing which questions to ask, they are insufficiently sensitive to the fact that different answers to the same question can have very different diagnosticities. This can lead subjects to overestimate or underestimate the information in the answers they receive. This phenomenon is demonstrated in two experiments using different kinds of inferences (category membership of individuals and composition of sampled populations). In combination with certain information-gathering tendencies, demonstrated in a third experiment, insensitivity to answer diagnosticity can contribute to a tendency toward preservation of the initial hypothesis. Results such as these illustrate the importance of viewing hypothesis-testing behavior as an interactive, multistage process that includes selecting questions, interpreting data, and drawing inferences.

Finding Useful Questions: On Bayesian Diagnosticity, Probability, Impact, and Information Gain.

“…To address this, the Vuma scenario (described in the above review of Sherman, 1986, andSlowiaczek et al, 1992, and in Appendix A) was simulated, with P(glom)=P(fizo)=0.5, and random feature probabilities, such as P(hulaWorn | glom). ("Random probabilities" denotes pseudorandom numbers independently sampled from a uniform distribution between [0,1].)…”

“…Several researchers explicitly make this point (Baron, 1985;Klayman & Ha, 1987;Klayman, 1987;Slowiaczek et al, 1992;Over & Jessop, 1998;Oaksford & Chater, 2003 (Skov & Sherman, 1986;Klayman & Ha, 1987, 1989Gorman, Stafford & Gorman, 1987;Devine, Hirt, & Gehrke, 1990; Finding useful questions 30 Slowiaczek et al, 1992;reviewed in Klayman, 1995). In some situations, strategies that reduce memory load (Costa-Gomes, Crawford, & Broseta, 2001) might also be used.…”

“…Most tasks have not specified this information, or have not obtained sequences of questions, and are therefore unable to address whether subjects' queries are sensitive to classconditional dependencies. In a review of work on perceptual information integration (not involving active sampling), Movellan and McClelland (2002) Each sampling norm will be discussed in the context of the planet Vuma scenario (Skov & Sherman, 1986;Slowiaczek et al, 1992). The task is to discover whether a novel creature is a glom or fizo, by asking whether it possesses a particular binary feature, such as wearing a hula hoop or not, or drinking iced tea or not.…”

“…Table 3 Features used by Skov and Sherman (1986). Slowiaczek et al (1992) 48% of gloms and 52% of fizos have feature A; 52% of gloms and 48% of fizos have feature B. Table 4 Features used by Slowiaczek et al (1992, experiment 3b) Tables in finding useful questions 56 Table 5 Re-analysis of experiment 4 in Baron et al (1988) Test : Subjects' ratings, and the conditional probabilities of a positive test given each disease, are adapted from Table 4 in "Heuristics and biases in diagnostic reasoning: II.…”

“…This notwithstanding, some researchers have implied that Bayesian diagnosticity is the only theoretically (normatively) defensible, or the only Bayesian, sampling norm. For instance, Slowiaczek et al (1992) stated "according to Bayes' theorem, the diagnosticity…”