The ultimate sampling dilemma in experience-based decision making.

Abstract

Computer simulations and two experiments are reported to delineate the ultimate sampling dilemma, which constitutes a serious obstacle to inductive inferences in a probabilistic world. Participants were asked to take the role of a manager who is to make purchasing decisions based on positive versus negative feedback about three providers in two different product domains. When information sampling (from a computerized data base) was over, they had to make inferences about actual differences in the data base from which the sample was drawn (e.g., about the actual superiority of different providers, or about the most likely origins of negatively valenced products). The ultimate sampling dilemma consists in a forced choice between two search strategies that both have their advantages and their drawbacks: natural sampling and deliberate sampling of information relevant to the inference task. Both strategies leave the sample unbiased for specific inferences but create errors or biases for other inferences. Many everyday decision problems rely on direct environmental-learning experience.Teachers' grading decisions are informed by observations of students' performance in different disciplines. Personnel selection relies on applicants' reactions to various tasks and interview topics. Or, consumer choices reflect the information acquired about brands or providers in different product domains. There appears to be a simple and straightforward way of optimizing such experience-based decisions: If only the learning process relies on a sufficiently large sample of observations, it must be possible to discern the optimal decision through optimal data selection (Oaksford & Chater, 2003).The learning task seems to have a clear-cut structure. For a consumer to make an optimal choice between alternative providers, it is only necessary to compare the quality feedback that is available for different providers in specific product domains. Granting that the feedback is reliable and accurately reflects the contingency between providers and product quality, figuring out the best provider, with the highest rate of positive evaluations, should be straightforward. The consumer's task should be easy to solve if only the differences between providers are strong enough and sufficient observations are available.The aim of the present investigation is to contest this seemingly plausible sketch of simple experience-based decision making. In fact, finding a generally correct solution to such clearly structured problems is fraught with huge difficulties. It is actually impossible, because every sample of observations about mundane decision problems entails the potential to be misleading under certain conditions. I refer to the "ultimate sampling dilemma" to highlight the fact that any reasonable sampling strategy, which serves to optimize one decision, produces a sampling bias with regard to other decisions informed by the same data. Illustration of the Ultimate Sampling DilemmaThat judgments and decisions depend crucially on the samples of...

Use of the recognition heuristic depends on the domain’s recognition validity, not on the recognition validity of selected sets of objects

“…In our two experiments, participants applied useless cues indifferently or did not detect useful ones. This could indeed be considered a case of meta-cognitive myopia (Fiedler, 2008(Fiedler, , 2012, a term that highlights shortsightedness in a variety of monitoring processes, for example, in erroneously accepting a sample as representative of the underlying domain (see also Hogarth & Soyer, 2015). However, maybe participants' behavior is not so ignorant and the bias not so obvious.…”

“…The caveat in this case is that one may select a strategy that does not actually yield the desired accuracy for the specific task at hand (or forgo use of a more accurate strategy), namely, if the set is not representative of the domain. Fiedler (2008) referred to this as Bmeta-cognitive myopia,^that is, the Bconspicuous reluctance to consider the history and the constraints imposed on the given stimulus sample^ (Fiedler, 2012, p. 42; see also Fiedler & Kutzner, in press;Hogarth & Soyer, 2015). Ignorantly presuming unbiased samples, however, could have detrimental consequences for choice performance.…”

Information Sampling, Belief Synchronization, and Collective Illusions

“…Like other information sampling model, our theory emphasizes the role of the environment and how it can sometimes produce systematically biased samples of information that affect judgments (see Fiedler & Juslin, 2006 for a review and Denrell & Le Mens, 2011;Eder, Fiedler & Hamm-Eder, 2011;Feiler, Tong & Larrick, 2013;Fiedler, 2008Fiedler, , 2011Fiedler, , 2012Henriksson, Elwin & Juslin, 2010;Le Mens & Denrell, 2011;Smith & Collins, 2009, for recent publications that adopt this perspective).…”

How Distinct are Intuition and Deliberation? An Eye-Tracking Analysis of Instruction-Induced Decision Modes

“…Different models that rely on automatic processes might be considered to account for intuition. These models range from mainly cognitive evidence accumulation (Busemeyer & Townsend, 1993), sampling (Dougherty, Gettys, & Ogden, 1999;Fiedler, 2008) or network models (Busemeyer & Johnson, 2004;Glöckner & Betsch, 2008b;Holyoak & Simon, 1999) to more affect-based approaches (Damasio, 1994;Finucane, Alhakami, Slovic, & Johnson, 2000). Furthermore, numerous theories modeling the interplay between intuitive and deliberate processes exist.…”