Limits in decision making arise from limits in memory retrieval

Abstract

Some decisions, such as predicting the winner of a baseball game, are challenging in part because outcomes are probabilistic. When making such decisions, one view is that humans stochastically and selectively retrieve a small set of relevant memories that provides evidence for competing options. We show that optimal performance at test is impossible when retrieving information in this fashion, no matter how extensive training is, because limited retrieval introduces noise into the decision process that cannot be overcome. One implication is that people should be more accurate in predicting future events when trained on idealized rather than on the actual distributions of items. In other words, we predict the best way to convey information to people is to present it in a distorted, idealized form. Idealization of training distributions is predicted to reduce the harmful noise induced by immutable bottlenecks in people's memory retrieval processes. In contrast, machine learning systems that selectively weight (i.e., retrieve) all training examples at test should not benefit from idealization. These conjectures are strongly supported by several studies and supporting analyses. Unlike machine systems, people's test performance on a target distribution is higher when they are trained on an idealized version of the distribution rather than on the actual target distribution. Optimal machine classifiers modified to selectively and stochastically sample from memory match the pattern of human performance. These results suggest firm limits on human rationality and have broad implications for how to train humans tasked with important classification decisions, such as radiologists, baggage screeners, intelligence analysts, and gamblers. categorization | cognitive modeling | uncertainty | diffusion modeling W hen making decisions, people often retrieve a limited set of items from memory (1-4). These retrieved items provide evidence for competing options. For example, a dark cloud may elicit memories of heavy rains, leading one to pack an umbrella instead of sunglasses. Likewise, when viewing an X-ray, a radiologist may retrieve memories of similar X-rays from other patients. Whether or not these other patients have a tumor may provide evidence for or against the presence of a tumor in the current patient. This process of sequential retrieval from memory, evidence accumulation, and final decision is formalized by diffusion models of choice and response time (Fig. 1). Diffusion models have successfully accounted for human learning and decision performance in a number of domains (1,5,6). These models accumulate noisy evidence retrieved from memory until a decision boundary is crossed. Wider boundaries imply higher accuracy and longer response times. For a given boundary, diffusion models optimally integrate evidence.However, according to the diffusion model, it is impossible to make optimal decisions within finite time (i.e., when the boundaries are finite). For probabilistic problems, such as determining whether a tumor is canc...

Improved classification of mammograms following idealized training.

“…In a simulated psychiatric diagnosis task, participants relied on easily accessible instances and their decisions were guided by the idiosyncratic properties of the training items (Young, Brooks & Norman, 2011). These results align closely with Giguère & Love's (2013) characterization of memory retrieval at the time of decision. To the extent that memory retrieval is limited to available (i.e., recent, familiar, and similar) instances, idealized training should improve test performance.…”

Visual category learning

“…Typically, a randomly selected exemplar from a randomly selected category is displayed to the subject on each trial. Alternatively, the selection of categories and exemplars from a category could be biased if a goal is to test the effects of frequency on category learning, or carefully controlled if a goal is to test the effects of staging of categories and exemplars on the speed, quality, and type of category learning . The particular details of each individual experimental trial of category learning can vary, sometimes as an intentional manipulation within an experiment, other times as a mere difference in convention across experimenters and laboratories.…”

“…Some recent work has examined how manipulating the presentation of category learning trials may optimize learning. Giguère and Love considered how people learn categories with overlapping distributions of category members. Because the distributions overlap, categorization is probabilistic, in the sense that the same object is sometimes associated with one category and other times associated with another category.…”

“…Alternatively, the selection of categories and exemplars from a category could be biased if a goal is to test the effects of frequency on category learning, 53 or carefully controlled if a goal is to test the effects of staging of categories and exemplars on the speed, quality, and type of category learning. [54][55][56] The particular details of each individual experimental trial of category learning can vary, sometimes as an intentional manipulation within an experiment, other times as a mere difference in convention across experimenters and laboratories. For example, each exemplar and its category label could be presented simultaneously, 57 or, each exemplar could be shown without a label and the subject could be asked to categorize it; subjects may 13,42 or may not 58 receive corrective feedback, that itself may vary.…”

Best-classifier feedback in diagnostic classification training.

“…For example, one could determine best-classifier responses based on average accuracy for each stimulus derived from a separate classification study. Such an approach could be used in concert with another recent line of research using idealized training sets to improve diagnostic performance (Giguère & Love, 2013; Hornsby & Love, 2014; Patil, Zhu, Kopeć, & Love, 2014). The approach used by Love and colleagues reduces the salience of ambiguous cases by removing them from the training set, thereby strengthening learning for the most critical aspects of each category.…”