A Model of Prefrontal Cortical Mechanisms for Goal-directed Behavior

Abstract

Many behavioral tasks require goal-directed actions to obtain delayed reward. The prefrontal cortex appears to mediate many aspects of goal-directed function. This paper presents a model of prefrontal cortex function emphasizing the influence of goal related activity on the choice of the next motor output. The model can be interpreted in terms of key elements of Reinforcement Learning theory. Different neocortical minicolumns represent distinct sensory input states and distinct motor output actions. The dynamics of each minicolumn include separate phases of encoding and retrieval. During encoding, strengthening of excitatory connections forms forward and reverse associations between each state, the following action, and a subsequent state, which may include reward. During retrieval, activity spreads from reward states throughout the network. The interaction of this spreading activity with a specific input state directs selection of the next appropriate action. Simulations demonstrate how these mechanisms can guide performance in a range of goal-directed tasks, and provide a functional framework for some of the neuronal responses previously observed in medial prefrontal cortex during performance of spatial memory tasks in rats.Numerous behavioral tasks involve goal-directed behavior based upon a delayed reward. For example, a rat in an instrumental task must generate lever presses to obtain food reward (Corbit and Balleine, 2003;Killcross and Coutureau, 2003;Wyble et al., 2004), and a rat in a T-maze must run down the stem of the maze to obtain food reward in one arm of the maze (Jung et al., 1998;Wood et al., 2000;Baeg et al., 2003;Ferbinteanu and Shapiro, 2003). Lesions of the prefrontal cortex cause impairments in goal-directed behavior (Fuster, 1995;Miller and Cohen, 2001;Corbit and Balleine, 2003;Killcross and Coutureau, 2003), and prefrontal units show firing dependent upon the association of cues and future responses (Miller, 2000). The model presented here addresses how goal directed behavior can be mediated by populations of neurons.An extensive theoretical framework termed Reinforcement Learning (Sutton, 1988;Sutton and Barto, 1998) describes how an agent can generate behaviors for delayed rewards in its environment. Current sensory input to the agent is represented by a "state" vector, and the output of the agent is represented by "actions" which alter the state vector (i.e., moving the agent to a different location). The selection of actions is guided by value functions (associating states with future reward) and state-action-value functions (associating actions in specific states with future reward). These functions are often learned using variants of temporal difference (TD) learning (Sutton, 1988;Sutton and Barto, 1998).Research has focused on the similarity between the error term in TD learning and the activity of dopaminergic neurons (Montague et al., 1996;Schultz et al., 1997). The basal ganglia have been proposed to provide circuitry for computation of TD learning (Houk et al., 1995). A...

An Integrate-and-fire Model of Prefrontal Cortex Neuronal Activity during Performance of Goal-directed Decision Making

“…This demonstrates how integrate-and-fire neurons can perform the circuit mechanism of action selection proposed in a more abstract model of the PFC (Hasselmo, 2005).…”

“…Our model replicates goal-directed behavior in a visual discrimination task based on a hypothesis about the functional connectivity of PFC circuits (Hasselmo, 2005). Behavioral responses and reward associations to visual cues are encoded in synaptic strengths between neuronal networks representing cortical minicolumns.…”

“…We show how populations of spiking neurons could interact to allow selection of specific actions based on the context of specific sensory input (states) and the desire for reward. Because activity in a specific minicolumn (Fuster, 2000) that represents such a state or action may play a role in different contexts that require its association with different state--action-state transitions, we presuppose separate populations of neurons within a minicolumn for input from and output to other minicolumns (Hasselmo, 2005). For example, the Go and Reward minicolumns in the experimental task fulfill such multiple roles, as shown in Figures 3 and 12A.…”

“…Integrate-and-fire neurons simulate the membrane potential response to the build-up of synaptic input over time and emit a spike when the potential crosses threshold. The model shows how integrate-and-fire neurons can perform the functions described in equations for a circuit model of the PFC (Hasselmo, 2005). The structure of the model was motivated by anatomical evidence suggesting the organization of neural circuits into minicolumns (Lund et al, 1993), cell assemblies of highly interconnected neurons found in the PFC.…”

Path planning versus cue responding: a bio-inspired model of switching between navigation strategies

“…This insures that graph nodes are only connected to their closest neighbors. Given the Planning Graph, the optimal path to the goal is determined by the activation-diffusion mechanism (Burnod 1991;Hasselmo 2005), based on the Dijkstra's algorithm for finding the shortest path between two nodes in a graph (Dijkstra 1959). More specifically, during goal planning, the Planning expert first determines its location using a position value and then calculates the direction toward the goal using goal value.…”