Insensitivity and Oversensitivity to Answer Diagnosticity in Hypothesis Testing

Abstract

Two experiments examined how people perceive the diagnosticity of different answers ("yes" and "no") to the same question. We manipulated whether the "yes" and the "no" answers conveyed the same amount of information or not, as well as the presentation format of the probabilities of the features inquired about. In Experiment 1, participants were presented with only the percentages of occurrence of the features, which most straightforwardly apply to the diagnosticity of "yes" answers. In Experiment 2, participants received in addition the percentages of the absence of features, which serve to assess the diagnosticity of "no" answers. Consistent with previous studies, we found that participants underestimated the difference in the diagnosticity conveyed by different answers to the same question. However, participants' insensitivity was greater when the normative (Bayesian) diagnosticity of the "no" answer was higher than that of the "yes" answer. We also found oversensitivity to answer diagnosticity, whereby participants valued as differentially diagnostic two answers that were normatively equal in terms of their diagnosticity. Presenting to participants the percentages of occurrence of the features inquired about together with their complements increased their sensitivity to the diagnosticity of answers. We discuss the implications of these findings for confirmation bias in hypothesis testing.Keywords: hypothesis testing; answer diagnosticity; insensitivity; oversensitivity; the featurepositive effect. INSENSITIVITY TO ANSWER DIAGNOSTICITY 3 Insensitivity and Oversensitivity to Answer Diagnosticity in Hypothesis TestingAn efficient evaluation of the impact of the information that we receive or we acquire is of critical importance in many everyday life situations as well as in many professional contexts.Consider, for example, the case of a deluded patient who interprets the appearance of a police car in the busy street where she is walking as a cue that the police is chasing her. In this case, the patient is disregarding the probability that the police is chasing another person and not her (Hemsley & Garety, 1986). In other words, the patient fails to value the evidential strength of the incoming evidence (the appearance of the police car) under hypotheses that are different from the one she is considering. Another way to misweigh evidence is failing to appreciate that the presence of an event might convey a different amount of information than its absence (e.g., Cherubini, Rusconi, Russo, & Crippa, in press; Fischhoff & Beyth-Marom, 1983;Rusconi, Crippa, Russo, & Cherubini, 2012; Rusconi, Marelli, Russo, D'Addario, & Cherubini, in press). For example, an investigator might fail to notice that the absence of a suspect's fingerprints from a crime scene can be more revealing than their presence if the suspect was familiar with the victim. Similarly, eyewitness nonidentifications might be at least as informative as identifications (e.g., Clark & Wells, 2008; Wells & Lindsay, 1980). The present artic...

Asking better questions: How presentation formats influence information search.

“…We tested the differences between presenting information with complement (e.g., 70% of turtles are Bayosian and 30% of turtles are Freqosian) and presenting information without complement (e.g., 70% of turtles are Bayosian). Rusconi and McKenzie (2013) have shown that the inclusion of complement information with the standard probability format improved sensitivity to the informativeness of an answer, which we hypothesized would also be helpful for making search decisions. This design feature was not manipulated in the visual formats, all of which intrinsically include complement information.…”

Being Honest and Acting Consistently: Boundary Conditions of the Negativity Effect in the Attribution of Morality

“…According to the authors, cue diagnosticity is an emergent property of cue frequency, and thus a rare cue is more informative than a common cue. Rather than diagnosticity as defined in the category learning theory, the latter conceptualization refers more squarely to the constructs of "surprisal" (Tribus, 1961), "self-information" in information theory (Rusconi, Crippa, Russo, & Cherubini, 2012), and the "rarity assumption" in the Bayesian reasoning literature (Anderson, 1990;McKenzie, 2006;McKenzie & Chase, 2012;McKenzie & Mikkelsen, 2000Oaksford & Chater, 1994;Rusconi & McKenzie, 2013), whereby "answers, or test outcomes, are diagnostic to the extent that they are rare or surprising" (McKenzie, 2006, p. 580). Future work should differentiate the terminology used to define the constructs of diagnosticity as emerged from the categorization literature and diagnosticity as defined in the information-theory field.…”

Evidence evaluation: Measure Z corresponds to human utility judgments better than measure L and optimal-experimental-design models.

“…For example, a person might ask of a new acquaintance, “Do you often organize parties?” to learn about her/his extroversion. The target of the query might answer “yes, I do,” and this response would convey a relatively different amount of information about her or his extroversion than would the answer “no, I do not.” In particular, the “yes” answer to the above question is relatively more diagnostic of the target’s extroversion (a person who often organizes parties is most likely extroverted) than is the “no” answer of the target’s introversion, because a person who does not often organize parties might still be extroverted (e.g., Brambilla, Rusconi, Sacchi, & Cherubini, 2011, Study 2; Cameron & Trope, 2004; Cherubini, Rusconi, Russo, Di Bari, & Sacchi, 2010; Rusconi & McKenzie, 2013; Rusconi, Sacchi, Toscano, & Cherubini, 2012; Sacchi, Rusconi, Bonomi, & Cherubini, 2013; Sacchi, Rusconi, Russo, Bettiga, & Cherubini, 2012; Trope & Liberman, 1996; Trope & Thompson, 1997). Therefore, the tester should revise her or his prior confidence about the target’s extroversion differently depending on whether she or he receives a “yes” or a “no” answer.…”

“…Therefore, the tester should revise her or his prior confidence about the target’s extroversion differently depending on whether she or he receives a “yes” or a “no” answer. A fair evaluation of the answers to a question (experiment results or test outcomes) is necessary for accurate revision of initial beliefs with respect to a single hypothesis or to multiple hypotheses (Rusconi & McKenzie, 2013; Slowiaczek, Klayman, Sherman, & Skov, 1992) and, eventually, for effective decision making. The present contribution investigated which, if any, of the different normative theories advanced by statisticians, epistemologists, philosophers of science, and psychologists best described human intuition about the value of obtained evidence.…”

“…There is a large Bayesian reasoning literature on how people evaluate evidence to make judgments (e.g., Beach, 1968; Cherubini, Rusconi, Russo, & Crippa, 2013; Cosmides & Tooby, 1996; Fischhoff & Beyth-Marom, 1983; Gigerenzer & Hoffrage, 1995; Hammerton, 1973; McKenzie, 1994; Rusconi, Crippa, Russo, & Cherubini, 2012; Rusconi, Marelli, Russo, D’Addario, & Cherubini, 2013; Rusconi & McKenzie, 2013; Slovic & Lichtenstein, 1971; Villejoubert & Mandel, 2002). The issue of whether people’s intuition about the value of obtained evidence align with theoretically optimal models has been the object of recent studies that noted the theoretical and empirical validity of two norms, namely, measures L and Z (e.g., Crupi, Tentori, & Gonzalez, 2007; Fitelson, 2001, 2006; Mastropasqua, Crupi, & Tentori, 2010; Tentori, Crupi, Bonini, & Osherson, 2007).…”