Optimal allocation of finite sampling capacity in accumulator models of multi-alternative decision making

Abstract

When facing many options, we narrow down our focus to very few of them. Although behaviors like this can be a sign of heuristics, they can actually be optimal under limited cognitive resources. Here we study the problem of how to optimally allocate limited sampling time to multiple options, modelled as accumulators of noisy evidence, to determine the most profitable one. We show that the effective sampling capacity of an agent increases with both available time and the discriminability of the options, and optimal policies undergo a sharp transition as a function of it. For small capacity, it is best to allocate time evenly to exactly five options and to ignore all the others, regardless of the prior distribution of rewards. For large capacities, the optimal number of sampled accumulators grows sub-linearly, closely following a power law for a wide variety of priors. We find that allocating equal times to the sampled accumulators is better than using uneven time allocations. Our work highlights that multi-alternative decisions are endowed with breadth-depth tradeoffs, demonstrates how their optimal solutions depend on the amount of limited resources and the variability of the environment, and shows that narrowing down to a handful of options is always optimal for small capacities.

Deep imagination is a close to optimal policy for planning in large decision trees under limited resources

“…Thus, by hard-wiring these strategies much of the overload caused by metareasoning [11,12,13] could be alleviated, allowing agents to use their finite resources for the tasks that change on a faster time scale. Finally, it is important to note that, in contrast to many experimental frameworks on binary choices or very low number of options [51,52,17,53] and games [45,54] where the number of actions is highly constrained by design, realistic decisions face too many immediate options to be all considered [21,22,23,26], and thus a first decision that cannot be deferred is how many of those to focus on in the first place [4,5,53,55]. All in all, the optimal BD tradeoffs that we have characterized here might play an important role even in cases that substantially depart from our modeling assumptions.…”

“…In all panels, points correspond to simulations (average over 10 6 runs) and solid lines correspond to theoretical predictions by Eqs. (4,5,6) and Eqs. (15,16,6) (Sec.3 of the Methods) for the homogeneous allocation case.…”

“…The dilemma that arises then is how to allocate limited search resources over large decision trees: should we consider many countries for our next vacation (breadth), at the cost of not evaluating very thoroughly any of them, or should we consider very few countries more deeply (depth), at the risk of missing the most exciting one? The above problem is one example of the so-called breadth-depth (BD) dilemma, important in tree search algorithms [1,2], optimizing menu designs [3], decision-making [4,5,6], knowledge management [7] and education [8].…”

“…Optimization of BD tradeoffs have been studied using the framework of infinitely manyarmed bandits and combinatorial multi-armed bandits where finite resources can be arbitrarily allocated among many options. These include one-shot infinitely many-armed Bernoulli [4] and Gaussian [5] bandits with compound actions, sequential infinitely many-armed Bernoulli bandits [28] and broader families thereof [29] with simple actions, and sequential combinatorial multiarmed bandits with compound actions [30]. These studies show that it is indeed optimal to ignore the vast majority of options while focusing sampling on a relatively small number of them that sublinearly scales with capacity [4,5].…”

“…These include one-shot infinitely many-armed Bernoulli [4] and Gaussian [5] bandits with compound actions, sequential infinitely many-armed Bernoulli bandits [28] and broader families thereof [29] with simple actions, and sequential combinatorial multiarmed bandits with compound actions [30]. These studies show that it is indeed optimal to ignore the vast majority of options while focusing sampling on a relatively small number of them that sublinearly scales with capacity [4,5]. However, the described optimal BD tradeoffs have been limited to trees of depth one, and thus how to balance breadth and depth search in decision trees remains an unresolved problem.…”