Deviation from the matching law reflects an optimal strategy involving learning over multiple timescales

Iigaya, Kiyohito; Ahmadian, Yashar; Sugrue, Leo P.; Corrado, Greg S.; Loewenstein, Yonatan; Newsome, William T.; Fusi, Stefano

doi:10.1038/s41467-019-09388-3

Cited by 41 publications

(93 citation statements)

References 55 publications

(99 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2F). In principle, prew allows them to predict upcoming reward, while loss count allows animals to track the local reward schedule, in agreement with previous studies (Aldiss & Davison, 1985;Gibbon et al, 1988;Herrnstein et al, 1997;Iigaya et al, 2019;Schneider & Davison, 2005;Sugrue et al, 2004). This suggests that animals were able to infer and make decisions using a hidden task variable (prew).…”

Section: Resultssupporting

confidence: 86%

“…The majority of foraging studies use variable interval (VI) reward schedules in which rewards become available after random intervals in a history-independent manner (Aldiss & Davison, 1985;Gibbon, Church, Fairhurst, & Kacelnik, 1988;Herrnstein et al, 1997;Iigaya et al, 2019;Schneider & Davison, 2005;Sugrue, Corrado, & Newsome, 2004). These studies estimate value using the local or global history of reward availability, which suggests that the valuable option is chosen intentionally.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Population coding of strategic variables during foraging in freely-moving macaques

Shahidi

Schrater

Wright

et al. 2019

Preprint

View full text Add to dashboard Cite

Animals forage within their environment to extract valuable resources at lowest cost. Previous studies have suggested that animals simply maximize the current flow of reward without predicting the future outcomes of their actions and that this recent reward rate is represented in various brain areas. To test this, we devised a foraging task in which the relevant reward dynamics were hidden from the animal, and wirelessly record population activity in dorsolateral prefrontal cortex (dlPFC) while monkeys forage freely in their environment. We discover that their brains indeed contain predictions of future rewards and plans of their next actions. By decoding the dynamic reward probability and the memory of recent outcomes from the dlPFC population response, we show that monkeys create an internal representation of reward dynamics. The decoded variables predicted animal’s subsequent actions better than either the true experimental variables or the raw neural responses. Our results suggest that the relevant task variables and behavioral decisions are dynamically encoded in prefrontal cortex during the time course of foraging.

show abstract

Section: Resultssupporting

confidence: 86%

mentioning

confidence: 99%

Population coding of strategic variables during foraging in freely-moving macaques

Shahidi

Schrater

Wright

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Both of these cases show that multiple timescales are important for memory consolidation, and memory is certainly a fundamental component when integration over time is required. In fact, our simulation studies show that the previously proposed model of synaptic metaplasticity [15,21] can capture some of the key aspects of our data [23] (See Figure S7).…”

Section: Discussionmentioning

confidence: 64%

“…(b). Deviation from the matching law: the fraction of choices allocated to one target is plotted as a function of the fraction of rewards that were obtained from the same target for different experimental days (top left Monkey F day 1-4, bottom left: day 21-24, top right Monkey G day 1-3, bottom right: day [21][22][23][24]. Each data point represents an estimate in a given block of trials, the solid line is a linear fit to the data.…”

Section: Discussionmentioning

confidence: 99%

Learning fast and slow: deviations from the matching law can reflect an optimal strategy under uncertainty

Iigaya

Ahmadian

Sugrue

et al. 2017

Preprint

Self Cite

View full text Add to dashboard Cite

Behavior which deviates from our normative expectations often appears irrational. A classic example concerns the question of how choice should be distributed among multiple alternatives. The so-called matching law predicts that the fraction of choices made to any option should match the fraction of total rewards earned from the option. This choice strategy can maximize reward in a stationary reward schedule. Empirically, however, behavior often deviates from this ideal. While such deviations have often been interpreted as reflecting 'noisy', suboptimal, decision-making, here we instead suggest that they reflect a strategy which is adaptive in nonstationary and uncertain environments. We analyze the results of a dynamic foraging task. Animals exhibited significant deviations from matching, and animals turned out to be able to collect more rewards when deviation was larger. We show that this behavior can be understood if one considers that * kiigaya@gatsby.ucl.ac.uk † sf2237@columbia.edu 1 peer-reviewed) is the author/funder. All rights reserved. No reuse allowed without permission.The copyright holder for this preprint (which was not . http://dx.doi.org/10.1101/141309 doi: bioRxiv preprint first posted online May. 24, 2017; animals had incomplete information about the environments dynamics. In particular, using computational models, we show that in such nonstationary environments, learning on both fast and slow timescales is beneficial. Learning on fast timescales means that an animal can react to sudden changes in the environment, though this inevitably introduces large fluctuations (variance) in value estimates. Concurrently, learning on slow timescales reduces the amplitude of these fluctuations at the price of introducing a bias that causes systematic deviations. We confirm this prediction in data -monkeys indeed solved the bias-variance tradeoff by combining learning on both fast and slow timescales. Our work suggests that multi-timescale learning could be a biologically plausible mechanism for optimizing decisions under uncertainty.

show abstract

“…In sequential decision-making tasks, normative models based on Bayesian principles (Bogacz, Brown, Moehlis, Holmes, & Cohen, 2006) reveal withinand across-trial accumulation processes that operate on two different time scales (Glaze, Kable, & Gold, 2015). The two time scales of value accumulation also appear in optimal models with foraging tasks that provide fixed (i.e., non-replenishing) reward probabilities (Iigaya et al, 2019), albeit with different timescales. Thus, fundamental questions emerge as to how individual neurons capture the multiplexed nature of decision-making phenomena.…”

Section: Implications For Neural Representationsmentioning

confidence: 99%

A normative account of choice history effects in mice and humans

Lopez-Yepez

Martin

Hulme

et al. 2020

Preprint

View full text Add to dashboard Cite

Choice history effects describe how future choices depend on the history of past choices. Choice history effects are typically framed as a bias rather than an adaptive phenomenon because the phenomenon generally degrades reward rates in experimental tasks. How-ever, in natural habitats, choices made in the past constrain choices that can be made in the future. For foraging animals, the probability of obtaining a reward in a given patch depends on the degree to which the animals have exploited the patch in the past. One problem with many experimental tasks that show choice history effects is that such tasks artificially decouple choice history from its consequences in regard to reward availability over time. To circumvent this, we used a variable interval (VI) reward schedule that reinstates a more natural contingency between past choices and future reward availability. By manipulating first- and second-order statistics of the environment, we dissociated choice history, reward history, and reaction times. We found that choice history effects reflect the growth rate of the reward probability of the unchosen option, reward history effects reflect environmental volatility, and reaction time reflects overall reward rate. By testing in mice and humans, we show that the same choice history effects can be generalized across species and that these effects are similar to those observed in optimal agents. Furthermore, we develop a new reinforcement learning model that explicitly incorporates choice history over multiple timescales into the decision process, and we examine its predictive adequacy in accounting for the associated behavioral data. We show that this new variant, known as the double trace model, has a higher predictive adequacy of choice data, in addition to better reward harvesting efficiency in simulated environments. Finally, we show that the choice history effects emerge in optimal models of foraging in habitats with diminishing returns, thus linking this phenomenon to a wider class of optimality models in behavioral ecology. These results suggests that choice history effects may be adaptive for natural contingencies between consumption and reward availability. This concept lends credence to a normative account of choice history effects that extends beyond its description as a bias.

show abstract

Deviation from the matching law reflects an optimal strategy involving learning over multiple timescales

Cited by 41 publications

References 55 publications

Population coding of strategic variables during foraging in freely-moving macaques

Population coding of strategic variables during foraging in freely-moving macaques

Learning fast and slow: deviations from the matching law can reflect an optimal strategy under uncertainty

A normative account of choice history effects in mice and humans

Contact Info

Product

Resources

About