Non-action Learning: Saving Action-Associated Cost Serves as a Covert Reward

Abstract

“To do or not to do” is a fundamental decision that has to be made in daily life. Behaviors related to multiple “to do” choice tasks have long been explained by reinforcement learning, and “to do or not to do” tasks such as the go/no-go task have also been recently discussed within the framework of reinforcement learning. In this learning framework, alternative actions and/or the non-action to take are determined by evaluating explicitly given (overt) reward and punishment. However, we assume that there are real life cases in which an action/non-action is repeated, even though there is no obvious reward or punishment, because implicitly given outcomes such as saving physical energy and regret (we refer to this as “covert reward”) can affect the decision-making. In the current task, mice chose to pull a lever or not according to two tone cues assigned with different water reward probabilities (70% and 30% in condition 1, and 30% and 10% in condition 2). As the mice learned, the probability that they would choose to pull the lever decreased (<0.25) in trials with a 30% reward probability cue (30% cue) in condition 1, and in trials with a 10% cue in condition 2, but increased (>0.8) in trials with a 70% cue in condition 1 and a 30% cue in condition 2, even though a non-pull was followed by neither an overt reward nor avoidance of overt punishment in any trial. This behavioral tendency was not well explained by a combination of commonly used Q-learning models, which take only the action choice with an overt reward outcome into account. Instead, we found that the non-action preference of the mice was best explained by Q-learning models, which regarded the non-action as the other choice, and updated non-action values with a covert reward. We propose that “doing nothing” can be actively chosen as an alternative to “doing something,” and that a covert reward could serve as a reinforcer of “doing nothing.”

Medial prefrontal cortex suppresses reward-seeking behavior with risk of positive punishment by reducing sensitivity to reward

“…We modified a two-tone lever-pull task for head-fixed mice that we previously reported 5 into a two-tone lever-pull task with the risk of two types of punishment: positive punishment (i.e., exposure to an air-puff) and negative punishment (i.e., omission of a reward) (Figures 1A and 1B). Two pure tones (6 and 10 kHz pure tone for 0.8–1.2 s), the reward-seeking action (pulling the lever after the go sound cue [pink noise] that followed the tone presentation), and the reward (a water drop delivered from a spout near the mouth) were common to both tasks.…”

“…To better understand how positive punishment (air-puff) and negative punishment (reward omission) affected the choice behavior (pull or non-pull) of the mouse, we constructed Q -learning models with a maximum of five parameters 5,24 to predict the choice behavior during the training sessions in the air-puff and omission tasks (Table S1; see STAR Methods for details). On the basis of our previous study 5 , we assumed that these tasks included two choices, pull and non-pull, and there were values of pulling the lever ( Q pull ) and non-pulling of the lever ( Q non-pull ) for both tone A and B trials in each task. The pull-choice probability in each trial was determined from the sigmoidal function of the difference between Q pull and Q non-pull in that trial.…”

“…To update Q -values, we introduced the following terms: a reward value term ( κ r ) that increased Q pull after the reward acquisition, a punishment term 25 ( κ p ) that reduced Q pull after the punishment was delivered (air-puff stimulus in the air-puff task and reward omission in the omission task), and a saving term ( Ψ , equal to a covert reward for non-pull) that increased Q non-pull after the lever was not pulled. We introduced the term Ψ because we previously found that Ψ well explained the decrease in the pull probability in tone A trials during training for a similar omission task 5 . We also added a learning rate ( α l ) and forgetting rate ( α f ), which represent the change rates of Q pull and Q non-pul 26,27 .…”

“…To address these issues, we examined the effect of pharmacological inactivation of mPFC on the probability of performing a lever pull that yields a reward under the risk of positive (air-puff stimulus) and negative (reward omission) punishments in head-fixed male mice. Although the head-fixed condition was more stressful for the mice than a free-moving condition, we head-fixed them using a method developed in a previous study 5 so that one-photon and two-photon calcium imaging could be used to detect the relevant cortical activity in future experiments 20-22 . We found that mPFC is necessary for behavioral inhibition under the risk of positive punishment, but not negative punishment.…”

Neuronal representations of reward-predicting cues and outcome history with movement in the frontal cortex

“…This dmFrC activity might be maintained by cortico-basal gangliathalamocortical pathways (Haber, 2014;Sesack and Grace, 2010). Recently, we found that, in head-fixed mice trained to pull a lever to obtain a reward in response to different tone cues assigned to different reward probabilities, the mice pulled the lever more frequently in response to the higher-reward-predicting cue than in response to the lower-reward-predicting cue (Tanimoto et al, 2020). If the neuronal activity in the relevant areas can be recorded and manipulated during this operant task, it should clarify how the dmFrC neurons integrate the information related to the reward-predicting cue from other brain areas and use it to select the appropriate action.…”