Interplay of approximate planning strategies

Abstract

Humans routinely formulate plans in domains so complex that even the most powerful computers are taxed. To do so, they seem to avail themselves of many strategies and heuristics that efficiently simplify, approximate, and hierarchically decompose hard tasks into simpler subtasks. Theoretical and cognitive research has revealed several such strategies; however, little is known about their establishment, interaction, and efficiency. Here, we use modelbased behavioral analysis to provide a detailed examination of the performance of human subjects in a moderately deep planning task. We find that subjects exploit the structure of the domain to establish subgoals in a way that achieves a nearly maximal reduction in the cost of computing values of choices, but then combine partial searches with greedy local steps to solve subtasks, and maladaptively prune the decision trees of subtasks in a reflexive manner upon encountering salient losses. Subjects come idiosyncratically to favor particular sequences of actions to achieve subgoals, creating novel complex actions or "options."planning | hierarchical reinforcement learning | memoization | pruning H umans and other animals often face complex tasks and environments in which they have to plan and execute long sequences of appropriate actions to achieve distant goals. One can represent the space of future actions and outcomes as a tree; such trees grow inordinately (often exponentially) large as a function of the length of the sequence (i.e., the depth of the tree). Rather little is definitively known about how this computational complexity is addressed. Work in the fields of reinforcement learning and artificial intelligence has suggested a number of heuristics that we describe below, namely, hacking, hierarchies, hoarding, and habitization (1-4). Various tasks have been designed to highlight individual heuristics; though how subjects generate and combine them without clear instruction has not been well characterized (however, see refs. 5 and 6).We previously designed a moderately deep planning problem to elicit a specific heuristic, in this case hacking or pruning of the decision tree (4). However, the task contains many of the elements that make choosing appropriately tricky in general. Thus, we closely examined the nature of, and individual differences between, the performance of subjects, shedding light on the interaction of heuristics in the self-generation of adaptive control when faced with a complex planning problem.Subjects had to plan a path through a maze so as to maximize their cumulative earnings. On each trial, they were placed in a random state and were asked to plan to a depth of 3, 4, or 5 moves ( Fig. 1 A and B). Because each depth involved a binary choice, planning to depths 3, 4, and 5 corresponded to choosing among a set of 8, 16, or 32 possible sequences. We previously found that the large immediate losses at particular branch points in the tree (the red transitions) encouraged subjects to eliminate possibly lucrative subbranches beneath those points ...

Evidence integration in model-based tree search

“…Despite such advances, however, comparatively little progress has so far been made toward characterizing the concrete process by which model-based decisions are reached, that is, the actual procedure through which a representation of the decision problem is translated into a choice (9,10,23). This situation contrasts sharply with what one finds in the literature on simple choice, where a number of detailed process models have been proposed.…”

“…In particular, it would be of interest to integrate ideas from hierarchical reinforcement learning, which have recently been applied to human decision making (10,(46)(47)(48)(49)(50), with the evidence accumulation framework.…”

“…In the most salient work along these lines, reward-based tree search has been conceptualized in terms provided by modelbased reinforcement learning, a computational framework in which reward-based decisions are based on an explicit model of the choice problem, a "cognitive map" of the decision tree itself (11). Under this rubric, recent work has illuminated several aspects of rewardbased tree search, providing an indication of how representations of decision problems are acquired and updated (12)(13)(14), where in the brain relevant quantities (e.g., cumulative rewards) are represented (15-17), and how model-based decision making interacts with simpler, habit-based choice mechanisms (15,(18)(19)(20)(21)(22).Despite such advances, however, comparatively little progress has so far been made toward characterizing the concrete process by which model-based decisions are reached, that is, the actual procedure through which a representation of the decision problem is translated into a choice (9,10,23). This situation contrasts sharply with what one finds in the literature on simple choice, where a number of detailed process models have been proposed.…”

“…Deciding among immediate actions, even at the first branch point, requires a consideration of all of the paths that unfold below. Decision making thus assumes the form of reward-based tree search (6)(7)(8)(9)(10).…”

Adaptive integration of habits into depth-limited planning defines a habitual-goal–directed spectrum

“…That is, simply pruning the decision tree after one level of planning and not switching to habitual values, as suggested in previous work (25,26), would estimate zero values for both first-stage choices, because there is no reward available at the first stage of the task. This strategy would predict indifference between the two first-stage choices, as opposed to the distinctive stay-probability pattern that is predicted by the plan-until-habit strategy and evident in our experimental data.…”