Navigating complex decision spaces: Problems and paradigms in sequential choice.

Abstract

To behave adaptively, we must learn from the consequences of our actions. Doing so is difficult when the consequences of an action follow a delay. This introduces the problem of temporal credit assignment. When feedback follows a sequence of decisions, how should the individual assign credit to the intermediate actions that comprise the sequence? Research in reinforcement learning provides two general solutions to this problem: model-free reinforcement learning and model-based reinforcement learning. In this review, we examine connections between stimulus-response and cognitive learning theories, habitual and goal-directed control, and model-free and model-based reinforcement learning. We then consider a range of problems related to temporal credit assignment. These include second-order conditioning and secondary reinforcers, latent learning and detour behavior, partially observable Markov decision processes, actions with distributed outcomes, and hierarchical learning. We ask whether humans and animals, when faced with these problems, behave in a manner consistent with reinforcement learning techniques. Throughout, we seek to identify neural substrates of model-free and model-based reinforcement learning. The former class of techniques is understood in terms of the neurotransmitter dopamine and its effects in the basal ganglia. The latter is understood in terms of a distributed network of regions including the prefrontal cortex, medial temporal lobes cerebellum, and basal ganglia. Not only do reinforcement learning techniques have a natural interpretation in terms of human and animal behavior, but they also provide a useful framework for understanding neural reward valuation and action selection.

Electrophysiological Responses to Feedback during the Application of Abstract Rules

“…To quantify the contributions of general and item-specific knowledge to choice behavior, we modeled the learning and decision-making process. To do so, we used temporal difference learning (Sutton & Barto, 1998), an influential technique in the field of artificial intelligence with strong ties to psychological theories of human and animal conditioning (Walsh & Anderson, in press) and physiological models of phasic dopamine responses (Schultz, 1998). Central to this technique is the idea that differences between actual and expected outcomes, or reward prediction errors, provide teaching signals.…”

Toward a Functional View of the p Factor in Psychopathology

“…It is characterized by many theorists as being the more basic, and as being especially responsive to emotions (e.g., Metcalfe & Mischel, 1999; Strack & Deutsch, 2004). Sometimes the term model-free learning is used to refer to the way in which this mode of function acquires information over time, reflecting the idea that its learning follows from an accumulation of associations (e.g., Daw, Niv, & Dayan, 2005; Dayan, 2008; Dolan & Dayan, 2013; Otto, Gershman, Markman, & Daw, 2013; Walsh & Anderson, 2014). Functioning in this mode may be said to reflect habits (e.g., Ouellette & Wood, 1998) or responsiveness to cues of the moment that trigger automatic responses.…”

Evaluating the Theoretic Adequacy and Applied Potential of Computational Models of the Spacing Effect

“…Some psychology models focus only on learning (e.g., reinforcement learning; Walsh & Anderson, 2014) or forgetting (e.g., mathematical forgetting functions; Rubin & Wenzel, 1996). This is in contrast to account in great detail for one or a small number of findings.…”

“…This is in contrast to account in great detail for one or a small number of findings. Some psychology models focus only on learning (e.g., reinforcement learning; Walsh & Anderson, 2014) or forgetting (e.g., mathematical forgetting functions; Rubin & Wenzel, 1996). Yet these are ongoing, dynamic, competing processes.…”