Beyond dichotomies in reinforcement learning

Abstract

Reinforcement learning (RL) is a framework of particular importance to psychology, neuroscience, and machine learning. Interactions between these fields, as promoted through the common hub of RL, has facilitated paradigm shifts relating multiple levels of analysis within a singular framework (e.g dopamine function). Recently, more sophisticated RL algorithms have been incorporated to better account for human learning, and in particular its oft documented reliance on two separable systems. However, along with many benefits, this dichotomous lens can distort questions, and may contribute to an unnecessarily narrow perspective on learning and decision making. Here we outline some of the consequences that come from over-confidently mapping algorithms, such as model-based vs. model-free RL, with putative cognitive processes. We argue that the field is well positioned to move beyond simplistic dichotomies, and we propose a means of re-focusing research questions toward the rich and complex components that comprise learning and decision making.

Executive function supports single-shot endowment of value to arbitrary transient goals

“…In addition, successful performance in the instructed goal condition was associated with increased connectivity between prefrontal cortex and reward-sensitive regions. These findings are consistent with a flexible neural currency for corticostriatal instrumental learning (Daniel & Pollmann, 2014), and challenge sharp dichotomies between goal-directed and instrumental forms of learning (Collins & Cockburn, 2020).…”

“…Our results also add a complication to a strict dichotomy between goal-directed and instrumental components of decision-making (Collins & Cockburn, 2020; Dickinson & Balleine, 1994; (Doll et al, 2012). In our study, we observed essentially overlapping neural signatures for single-shot goal outcomes and secondary reinforcer outcomes, the latter reflecting conventional model-free learning feedback (Figure 3).…”

“…This hypothesis conflicts with a common dichotomy in the reinforcement learning field that separates goal-directed and instrumental behavior (Collins & Cockburn, 2020;Dickinson & Balleine, 1994). According to this model, learning from novel abstract, instructed goals arises from statistical learning of transitions between states of the world, and planning over these learned transitions to achieve goals; indeed, this process has been behaviorally and physiologically dissociated from standard RL, and linked to prefrontal executive function (Daw et al, 2011;Gläscher et al, 2010).…”

Mesolimbic dopamine adapts the rate of learning from action

“…These results suggest that the dopamine-dependent attribution of motivational value to cues 12,[71][72][73] is at least partially dissociable from the regulation of policy learning within the same mesolimbic circuits, suggesting that they may be complimentary building blocks for animal learning 28 .…”

“…Even though it has received relatively less attention in neuroscience, there are a number of reasons why it is important to explore the potential of direct policy learning to provide "computational and mechanistic primitives" 28 that account for aspects of dopamine function, especially in the context of novel task acquisition by animals. For one, behavioral adaptation during the initial acquisition of tasks is meaningfully variable across individuals 29 .…”