Evidence integration in model-based tree search

Abstract

Research on the dynamics of reward-based, goal-directed decision making has largely focused on simple choice, where participants decide among a set of unitary, mutually exclusive options. Recent work suggests that the deliberation process underlying simple choice can be understood in terms of evidence integration: Noisy evidence in favor of each option accrues over time, until the evidence in favor of one option is significantly greater than the rest. However, real-life decisions often involve not one, but several steps of action, requiring a consideration of cumulative rewards and a sensitivity to recursive decision structure. We present results from two experiments that leveraged techniques previously applied to simple choice to shed light on the deliberation process underlying multistep choice. We interpret the results from these experiments in terms of a new computational model, which extends the evidence accumulation perspective to multiple steps of action.reward-based decision making | drift-diffusion model | reinforcement learning I magine a customer standing at the counter in an ice cream shop, deliberating among the available flavors. Such a scenario exemplifies "simple choice," a decision situation in which the objective is to select among a set of individual, immediate outcomes, each carrying a different reward. Simple choice, in this sense, has provided a convenient focus for a great deal of work in behavioral economics and decision neuroscience (1-5). However, it would be an obvious mistake to treat it as an exhaustive model of rewardbased decision making. The decisions that arise in everyday life are of course often more complicated. One important difference, among others, is that everyday decisions tend to involve sequences of actions and outcomes.As an illustration, let us return to the ice cream customer, picturing him at a point slightly earlier in the day, exiting his home in quest of something sweet. Upon reaching the sidewalk, he faces a decision between heading left toward the ice cream shop, or heading right toward a frozen yogurt shop. If he wishes to fully evaluate the relative appeal of these two options, he must answer a second set of questions: Which flavor would he choose in each shop? Furthermore, it may be relevant for him to consider more immediate consequences of the left-right decision. For example, the leftward path might pass by a bank, allowing him to deposit a check along his way, whereas the rightward path might lead by the post office, giving him the opportunity to mail a package.Rather than selecting among individual and immediate outcomes, the decision maker in this scenario finds himself at the root of a decision tree (Fig. 1A), with nodes corresponding to value-laden outcomes or states, and edges corresponding to choiceinduced state transitions. Deciding among immediate actions, even at the first branch point, requires a consideration of all of the paths that unfold below. Decision making thus assumes the form of reward-based tree search (6-10).Note that decision makin...

The Neural Representation of Prospective Choice during Spatial Planning and Decisions

“…One disadvantage of our task is the inability to probe the time scale of plan formation and implementation in novel environments, particularly when choice accuracy and RT are influenced differently by path length differences. Most planning studies test after extensive training and are biased towards action-by-action evaluation without the need to maintain prior choices [3–4,48–50]. With extensively trained choices, the neural computations leading to increased decision implementation/RT are well studied [51–52].…”

Reinforcement Learning and Episodic Memory in Humans and Animals: An Integrative Framework

“…Their prevalence can be manipulated situationally (Otto et al, 2013a,b); varies across individuals (e.g., with symptoms of compulsive disorders such as drug abuse; Gillan et al, 2016); and tracks “prospective” representation of future states measured in fMRI at choice time (consistent with choice-time evaluation via mental simulation; Doll et al, 2015). Research with elaborated multi-step decision tasks has also begun to shed light on computational shortcuts by which the brain manages to compute the expected reward (Dezfouli and Balleine, 2013; Diuk et al, 2013; Huys et al, 2015; Cushman and Morris, 2015; Solway & Botvinick, 2015). …”

Planning and navigation as active inference

“…active inference) when plausible constraints are applied: see also Solway and Botvinick (2015). These constraints range from general principles to specific constraints that must be respected by real agents or sentient creatures.…”