Disentangling the Roles of Approach, Activation and Valence in Instrumental and Pavlovian Responding

Abstract

Hard-wired, Pavlovian, responses elicited by predictions of rewards and punishments exert significant benevolent and malevolent influences over instrumentally-appropriate actions. These influences come in two main groups, defined along anatomical, pharmacological, behavioural and functional lines. Investigations of the influences have so far concentrated on the groups as a whole; here we take the critical step of looking inside each group, using a detailed reinforcement learning model to distinguish effects to do with value, specific actions, and general activation or inhibition. We show a high degree of sophistication in Pavlovian influences, with appetitive Pavlovian stimuli specifically promoting approach and inhibiting withdrawal, and aversive Pavlovian stimuli promoting withdrawal and inhibiting approach. These influences account for differences in the instrumental performance of approach and withdrawal behaviours. Finally, although losses are as informative as gains, we find that subjects neglect losses in their instrumental learning. Our findings argue for a view of the Pavlovian system as a constraint or prior, facilitating learning by alleviating computational costs that come with increased flexibility.

The role of learning-related dopamine signals in addiction vulnerability

“…Following McClure et al (2003b), we have explained that this type of learning leads to representations in terms of model-free values, and that these capture key features of individual processing of motivational value, incentive salience assignment and sign-tracking. As such, it provides a framework within which neurobiology and behaviour relevant to addiction can be related in a computationally coherent manner (Redish et al, 2008;Dayan, 2009;Huys et al, 2013a), and forms one example of the application of computational neuroscience to psychiatric problems (Maia and Frank, 2011;Huys et al, 2011;Hasler, 2012;Montague et al, 2012;Huys et al, subm) However, much remains to be done. While the description of model-free learning and the neurobiological details of the circuits computing prediction errors advance rapidly, our understanding of the representations and computations underlying model-based reasoning remain poorly defined.…”

“…This is demonstrated in two types of Pavlovian-instrumental transfer (PIT), general and outcome-specific PIT. In both types of PIT, appetitive CSs enhance and aversive CSs suppress instrumental behaviours for other outcomes (Estes and Skinner, 1941;Rescorla and Solomon, 1967;Lovibond, 1983;Cardinal et al, 2002;Niv et al, 2007;Holmes et al, 2010;Talmi et al, 2008;Huys et al, 2011). In general PIT, a stimulus that has been paired in a Pavlovian manner with one type of outcome (e.g.…”

“…The newest version of DSM, DSM-5, states that an important characteristic of substance use disorders is an underlying change in brain circuitry that may persist beyond detoxification, particularly in individuals with severe disorders, without identifying what the specific underlying processes or "changes" might be. This chapter focuses on novel theoretical approaches and computational models from machine learning and decision theory in the hope that they might lend new scientific rigour to addiction research (Huys et al, 2011;Hasler, 2012). One of the beauties of addiction research is that the reinforcing effects of drugs of abuse and the development of drug dependence can be easily modelled in animals with high validity; and that theoretical frameworks are at a rather advanced stage of development.…”

Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive.

“…Similarly, models that include Pavlovian to instrumental transfer (PIT) effects (e.g., Huys et al, 2011) account for the impact of stimulus values on the overall invigoration of action but do not predict differential effects on choice of positively versus negative valenced options. Berridge and Zhang's model (Zhang et al, 2009) tries to account for all learning and incentive effects through a single incentive mechanism, reflected by a parameter.…”

Aversive Pavlovian Control of Instrumental Behavior in Humans

“…Adaptive behavior depends on interactions between systems regulating affective versus rational, instrumental control (Huys et al, 2011;Evans, 2008;Daw, Niv, & Dayan, 2005). Many decision-making phenomena that appear irrational, such as the framing effect (Tversky & Kahneman, 1981) and the optimism bias (Sharot, Riccardi, Raio, & Phelps, 2007;Weinstein, 1980), may reflect Pavlovian impact of affective cues on instrumental behavior (Dayan & Huys, 2008;Dayan, Niv, Seymour, & Daw, 2006).…”