Emergence of complex dynamics of choice due to repeated exposures to extinction learning

Abstract

Extinction learning, the process of ceasing an acquired behavior in response to altered reinforcement contingencies, is essential for survival in a changing environment. So far, research has mostly neglected the learning dynamics and variability of behavior during extinction learning and instead focused on a few response types that were studied by population averages. Here, we take a different approach by analyzing the trial-by-trial dynamics of operant extinction learning in both pigeons and a computational model. The task involved discriminant operant conditioning in context A, extinction in context B, and a return to context A to test the context-dependent return of the conditioned response (ABA renewal). By studying single learning curves across animals under repeated sessions of this paradigm, we uncovered a rich variability of behavior during extinction learning: (1) Pigeons prefer the unrewarded alternative choice in one-third of the sessions, predominantly during the very first extinction session an animal encountered. (2) In later sessions, abrupt transitions of behavior at the onset of context B emerge, and (3) the renewal effect decays as sessions progress. While these results could be interpreted in terms of rule learning mechanisms, we show that they can be parsimoniously accounted for by a computational model based only on associative learning between stimuli and actions. Our work thus demonstrates the critical importance of studying the trial-by-trial dynamics of learning in individual sessions, and the unexpected power of "simple" associative learning processes. Significance StatementOperant conditioning is essential for the discovery of purposeful actions, but once a stimulus-response association is acquired, the ability to extinguish it in response to altered reward contingencies is equally important. These processes also play a fundamental role in the development and treatment of pathological behaviors such as drug addiction, overeating and gambling. Here we show that extinction learning is not limited to the cessation of a previously reinforced response, but also drives the emergence of complex and variable choices that change from learning session to learning session. At first sight, these behavioral changes appear to reflect abstract rule learning, but we show in a computational model that they can emerge from "simple" associative learning.

Learning shifts the preferred theta phase of gamma oscillations in CA1

“…Each correct, omission, and incorrect response was encoded as 1, 0, and −1, respectively. Thus, the cumulative curve's positive, or a negative slope, reveals a preference for the correct or incorrect arm, respectively (Donoso et al, 2021). In contrast, a flat curve develops when random choices and/or omissions are expressed.…”

“…To investigate the dynamics of the behavior over the three experimental days, we plotted the cumulative sum of the behavioral responses as a function of each trial as previously described (Donoso et al, 2021). Briefly, the behavioral response in each trial was encoded as either 1 if it was the correct response, À1 for the alternative response, that is, choosing the incorrect arm, 0 for omission and À2 for moving back without making any decision.…”

The functional role of episodic memory in spatial learning

“…While in hippocampal research it is fairly common to report learning curves, these are generally averaged over animals and/or blocks of learning. However, such averaging can produce misleading results (Gallistel et al, 2004; Smith et al, 2004) For instance, Cheng and Sabes (2006, 2007) show that studying the dynamics of learning reveals an enhanced picture of the process of adaptation, and Donoso et al (2021) uncovered a large variability of behavior in extinction learning and renewal by focusing on individual learning curves. None of these findings could have been revealed by simply comparing the differences in performance before and after learning.…”