Evidence evaluation: Measure Z corresponds to human utility judgments better than measure L and optimal-experimental-design models.

Abstract

Evidence evaluation is a crucial process in many human activities, spanning from medical diagnosis to impression formation. The present experiments investigated which, if any, normative model best conforms to people's intuition about the value of the obtained evidence. Psychologists, epistemologists, and philosophers of science have proposed several models to account for people's intuition about the utility of the obtained evidence with respect either to a focal hypothesis or to a constellation of hypotheses. We pitted against each other the so called optimal-experimental-design models (i.e., Bayesian diagnosticity, log 10 diagnosticity, information gain, Kullback-Leibler distance, probability gain, and impact) and measures L and Z to compare their ability to describe humans' intuition about the value of the obtained evidence. Participants received words-andnumbers scenarios concerning two hypotheses and binary features. They were asked to evaluate the utility of "yes" and "no" answers to questions about some features possessed in different proportions (i.e., the likelihoods) by two types of extraterrestrial creatures (corresponding to two mutually exclusive and exhaustive hypotheses). Participants evaluated either how an answer was helpful or how an answer decreased/increased their beliefs with respect either to a single hypothesis or to both hypotheses. We fitted mixed-effects models and we used the Akaike information criterion (AIC) and the Bayesian information criterion (BIC) values to compare the competing models of the value of the obtained evidence. Overall, the experiments showed that measure Z was the best-fitting model of participants' judgments of the value of obtained answers. We discussed the implications for the human hypothesis-evaluation process.

Generalized Information Theory Meets Human Cognition: Introducing a Unified Framework to Model Uncertainty and Information Search

“…(2015), Meder and Nelson (2012), Nelson et al (2010), and Rusconi, Marelli, D'Addario, Russo, and Cherubini (2014), while Crupi, Tentori, and Lombardi (2009) relied on R Error in their critical analysis of so-called pseudodiagnosticity (also see Crupi & Girotto, 2014;Tweeney, Doherty, & Kleiter, 2010). 5 The idea of ln t is often credited to Tsallis for his work in generalized thermodynamics (see Tsallis, 1988Tsallis, , 2011.…”

“…Having a plurality of related measures available is indeed an important opportunity. It prompts thorough investigation of the features of alternative options and their relationships (e.g., Crupi, Chater, & Tentori, ; Huber & Schmidt‐Petri, ; Nelson, , ), it can provide a rich source of tools for both theorizing and the design of new experimental investigations (e.g., Rusconi et al., ; Schupbach, ; Tentori, Crupi, Bonini, & Osherson, ), and it makes it possible to tailor specific models to varying tasks and contexts within an otherwise coherent approach (e.g., Crupi & Tentori, ; Dawid & Musio, ; Oaksford & Hahn, ).…”

What can the conjunction fallacy tell us about human reasoning?

“…The explanation of the CF provided in the previous section suggests that common probability errors can be determined by a preponderance of evidential reasoning over probabilistic reasoning. In this regard, it is worth noting that people's judgments of evidential impact have been reported to be accurate, both when applied to the evaluation of abstract arguments concerning, for example, urns and balls of different colors (Tentori et al, 2007), and in everyday tasks that require participants to quantify the impact of uncertain evidence (Mastropasqua et al, 2010) or the value of evidence with regard to competing hypotheses (Crupi et al, 2009;Rusconi et al, 2014). These results are consistent with those from the category-based induction literature, according to which adults, and even children as young as 5, when evaluating argument strength, follow popular principles of evidential impact, such as the similarity between premises and conclusion and the diversity of premises (Heit and Hahn, 2001;Lo et al, 2002;Lopez et al, 1992;Osherson et al, 1990;Zhong et al, 2014).…”

Towards a theory of heuristic and optimal planning for sequential information search

“…In this section we consider a two-category, binary-feature scenario ("Planet Vuma") that is one of the most frequently used tasks in psychological research on human information selection (Meder & Nelson, 2012;Nelson, 2005a;Nelson et al, 2010;Rusconi et al, 2014;Skov & Sherman, 1986;Slowiaczek et al, 1992;Wu et al, 2017). In the Planet Vuma scenario, the task is to categorize a fictitious alien into one of two species (hypotheses h 1 and h 2 ) by querying their (binary) features.…”