Absence of “Warm-Up” during Active Avoidance Learning in a Rat Model of Anxiety Vulnerability: Insights from Computational Modeling

Abstract

Avoidance behaviors, in which a learned response causes omission of an upcoming punisher, are a core feature of many psychiatric disorders. While reinforcement learning (RL) models have been widely used to study the development of appetitive behaviors, less attention has been paid to avoidance. Here, we present a RL model of lever-press avoidance learning in Sprague-Dawley (SD) rats and in the inbred Wistar Kyoto (WKY) rat, which has been proposed as a model of anxiety vulnerability. We focus on “warm-up,” transiently decreased avoidance responding at the start of a testing session, which is shown by SD but not WKY rats. We first show that a RL model can correctly simulate key aspects of acquisition, extinction, and warm-up in SD rats; we then show that WKY behavior can be simulated by altering three model parameters, which respectively govern the tendency to explore new behaviors vs. exploit previously reinforced ones, the tendency to repeat previous behaviors regardless of reinforcement, and the learning rate for predicting future outcomes. This suggests that several, dissociable mechanisms may contribute independently to strain differences in behavior. The model predicts that, if the “standard” inter-session interval is shortened from 48 to 24 h, SD rats (but not WKY) will continue to show warm-up; we confirm this prediction in an empirical study with SD and WKY rats. The model further predicts that SD rats will continue to show warm-up with inter-session intervals as short as a few minutes, while WKY rats will not show warm-up, even with inter-session intervals as long as a month. Together, the modeling and empirical data indicate that strain differences in warm-up are qualitative rather than just the result of differential sensitivity to task variables. Understanding the mechanisms that govern expression of warm-up behavior in avoidance may lead to better understanding of pathological avoidance, and potential pathways to modify these processes.

Testing the role of reward and punishment sensitivity in avoidance behavior: A computational modeling approach

“…7 ) and in order to maintain a simple framework, we proceed with the model configuration with fewer free parameters, where (α + )=(α − ) and (ε + )=(ε − ). Such configuration is consistent with other related models (e.g., Moutoussis et al 2008 ; Myers et al 2014 ) and proposes a comprehensive mechanism that is based solely on individual differences in outcome sensitivities. We are specifically interested to test whether the same computational mechanism that was proposed to underlie differences in avoidance acquisition could be applied to address differences in extinction learning.…”

“…Unless mentioned otherwise, all simulation results represent the average of 100 simulations, which is comparable to number of participants reported in recent studies using this task ( Sheynin et al 2014a ; Sheynin et al 2014b ). For simplicity, we first set (α + )=(α − ) and (ε + )=(ε − ), as often done with similar models (e.g., Moutoussis et al 2008 ; Myers et al 2014 ). Then, we show how specific manipulations of these LRs affect model behavior.…”

“…It should also be noted that the current work used a parameter-tuning approach to test specific hypotheses regarding the role of distinct reward and punishment sensitivities in avoidance behavior. While a few models of avoidance behavior have been previously reported ( Johnson et al 2001 ; Moutoussis et al 2008 ; Maia 2010 ; Smith et al 2004 ; Schmajuk and Zanutto 1997 ), the current model together with another parameter-tuning model of rodent avoidance behavior ( Myers et al 2014 ) represent the first attempts to use computational techniques to understand individual differences in this behavior. The parameter-tuning approach is often used to test the effect of specific parameters on the model behavior (e.g., Tables 1–2 in O'Reilly and Frank 2006 ), to simulate group differences (e.g., in Parkinson’s disease; Moustafa and Gluck 2011 ) and is especially advantageous when addressing a priori hypotheses concerning possible abnormal processes in the studied subjects (for review, see Fig.…”

“…By developing computational models that simulate the observed behavior, researchers can shed light on previous findings, as well as generate new hypotheses that would drive future empirical work. Traditionally, computational models of avoidance learning have used reinforcement learning algorithms (RL; Johnson et al 2001 ; Moutoussis et al 2008 ; Maia 2010 ; Smith et al 2004 ; Schmajuk and Zanutto 1997 ; Myers et al 2014 ), in which an “agent” (model) learns by trial and error to maximize reward and minimize punishment ( Sutton and Barto 1998 ). Specifically, if a response to a stimulus results in a reward (or better-than-expected outcome), the probability of repeating this response in the future is increased.…”

“…Here, we adapt a computational modeling approach that has been previously used to simulate common rodent conditioned avoidance paradigms (e.g., Maia 2010 ; Myers et al 2014 ). Specifically, we use a RL network model with an actor-critic architecture and apply it to the findings from recent studies with the computer-based spaceship task ( Sheynin et al 2014a ; Sheynin et al 2014b ).…”