Time elapsed between choices in a probabilistic task correlates with repeating the same decision

Abstract

Reinforcement learning makes an action that yielded a positive outcome more likely to be taken in the future. Here, we investigate how the time elapsed from an action affects subsequent decisions. Groups of C57BL6/J mice were housed in IntelliCages with access to water and chow ad libitum, and they were able to access bottles with a reward in the form of a saccharin solution (0.1% w/v), alcohol (4% w/v) or a mixture of the two. The probability of receiving a reward in two of the cage corners changed between 0.9 and 0.3 every 48 h over a period of ~33 days. We observed that in most animals, the odds of repeating the choice of a corner were increased if that choice was previously rewarded. Interestingly, in many cases, the time elapsed from the previous choice also increased the probability of repeating the choice, irrespective of the previous outcome. Behavioral data were fitted with a series of reinforcement learning models based on Q-learning. Then, extensions were introduced to account for attention or task complexity components or to directly include the interval length in order to simulate the decay of the expected value or to increase the probability of repeating the same choice. We find that introducing an interval-dependent adjustment to repeating the same choice allowed for the best prediction of the observed behavior, and the size of this effect may differ depending on the type of reward offered.