Modeling the influence of working memory, reinforcement, and action uncertainty on reaction time and choice during instrumental learning

Abstract

What determines the speed of our decisions? Various models of decision-making have focused on perceptual evidence, past experience, and task complexity as important factors determining the degree of deliberation needed for a decision. Here, we build on a sequential sampling decision-making framework to develop a new model that captures a range of reaction time (RT) effects by accounting for both working memory and instrumental learning processes. The model captures choices and RTs at various stages of learning, and in learning environments with varying complexity. Moreover, the model generalizes from tasks with deterministic reward contingencies to probabilistic ones. The model succeeds in part by incorporating prior uncertainty over actions when modeling RT. This straightforward process model provides a parsimonious account of decision dynamics during instrumental learning and makes unique predictions about internal representations of action values.

A new model of decision processing in instrumental learning tasks

“…Both RL-DDM and RL-ARD models tended to underestimate RTs and choice accuracy in the early trial bins in the accuracy emphasis condition. As in experiment 1, working memory may have contributed to the accurate but slow responses in the first trial bin for the accuracy condition (Collins and Frank, 2018, 2012b; McDougle and Collins, 2020).…”

“…SARSA assumes learning from feedback, and so cannot explain such high early accuracies. Working memory processes could have aided performance in the easiest condition, since the total number of stimuli pairs was limited and feedback was quite reliable, making it relatively easy to remember correct-choice options (Collins and Frank, 2018, 2012a; McDougle and Collins, 2020).…”

“…However, in the RL-EAMs, the posterior distributions for these Q-values consistently converged on 0, which was therefore subsequently used as a fixed value for all results reported here. For the soft-max fits, they were set to 0.5 as often used in reinforcement learning models of two-choice tasks (Apps et al, 2015; Collins and Frank, 2018, 2012b; Fontanesi et al, 2019a; McDougle and Collins, 2020; Pedersen and Frank, 2020). Including the initial Q-values as a free parameter in the soft-max models of experiment 2 led to the same conclusions.…”

“…Recent advances (Fontanesi et al, 2019a, 2019b; Luzardo et al, 2017; McDougle and Collins, 2020; Miletić et al, 2020; Millner et al, 2018; Pedersen et al, 2017; Pedersen and Frank, 2020; Sewell et al, 2019; Sewell and Stallman, 2020; Shahar et al, 2019; Turner, 2019) have emphasized how both modelling traditions can be combined in joint models of reinforcement learning (RL) and evidence-accumulation decision-making processes, providing mutual benefits for both fields. Combined models generally propose that value-based decisionmaking and learning interact as follows: For each decision a subject gradually accumulates evidence for each choice option by sampling from a distribution of memory representations of the subjective value (or expected reward ) associated with each choice option (known as Q-values ).…”

The role of executive function in shaping reinforcement learning

“…Moreover, this process appears to be negatively related to the use of goal-directed planning strategies, suggesting that EF enables RL to focus in on the task-relevant action space. Similarly, recent modeling work suggests that a stateless form of action values -that is, action values computed independently of any specific context -can exert an influence on both choices and reaction times [43], particularly when cognitive load is high. One hypothetical consequence of independently learning over the action dimension is that when executive functions are disrupted or taxed, and thus cannot properly conjoin states and actions, action values may be learned in a vacuum.…”