Reversal Learning in Humans and Gerbils: Dynamic Control Network Facilitates Learning

Abstract

Biologically plausible modeling of behavioral reinforcement learning tasks has seen great improvements over the past decades. Less work has been dedicated to tasks involving contingency reversals, i.e., tasks in which the original behavioral goal is reversed one or multiple times. The ability to adjust to such reversals is a key element of behavioral flexibility. Here, we investigate the neural mechanisms underlying contingency-reversal tasks. We first conduct experiments with humans and gerbils to demonstrate memory effects, including multiple reversals in which subjects (humans and animals) show a faster learning rate when a previously learned contingency re-appears. Motivated by recurrent mechanisms of learning and memory for object categories, we propose a network architecture which involves reinforcement learning to steer an orienting system that monitors the success in reward acquisition. We suggest that a model sensory system provides feature representations which are further processed by category-related subnetworks which constitute a neural analog of expert networks. Categories are selected dynamically in a competitive field and predict the expected reward. Learning occurs in sequentialized phases to selectively focus the weight adaptation to synapses in the hierarchical network and modulate their weight changes by a global modulator signal. The orienting subsystem itself learns to bias the competition in the presence of continuous monotonic reward accumulation. In case of sudden changes in the discrepancy of predicted and acquired reward the activated motor category can be switched. We suggest that this subsystem is composed of a hierarchically organized network of dis-inhibitory mechanisms, dubbed a dynamic control network (DCN), which resembles components of the basal ganglia. The DCN selectively activates an expert network, corresponding to the current behavioral strategy. The trace of the accumulated reward is monitored such that large sudden deviations from the monotonicity of its evolution trigger a reset after which another expert subnetwork can be activated—if it has already been established before—or new categories can be recruited and associated with novel behavioral patterns.

Machine learning identifies the dynamics and influencing factors in an auditory category learning experiment

“…These results contribute important information for improving formal models of category learning in the auditory domain. For example, the preferences in perceptual salience and motor response output can be considered in recurrent neural networks 13 by changing connectivity weights in the sensory imput or motor output layers of the network whereas in cognitive modelling using ACT-R 11 , initial activation values of chunks for rule selection can www.nature.com/scientificreports www.nature.com/scientificreports/ be adapted accordingly. The fact that current versions of these models do not make a priori assumptions about preferences of human learners may explain their lower average performance or need for many more trials as compared to human learners.…”

“…Behavioural performance of subjects in such category learning tasks has been the focus of several formal models (e.g. [4][5][6][7][8][9][10][11][12][13] ). However, researchers involved in the development of such computational models have recently come to the conclusion that existing models cannot fully explain humans' rule-based category learning 14 .…”

“…Answers to these questions will improve recent formal models of auditory category learning 11,13 regarding potential differences in salience between sound features and preferences in motor response behaviour. Q3 is specifically motivated by the wish to optimise experiment duration, avoiding fatigue and its potential side effects on performance.…”

From Crystallized Adaptivity to Fluid Adaptivity in Deep Reinforcement Learning — Insights from Biological Systems on Adaptive Flexibility

“…[5]), established learning algorithms used in engineering usually fail to show the level of fluid adaptivity that characterizes biological agents (e.g. [12], [19]). Previous studies have already established that significant performance improvements in evolutionary or learning algorithms can be achieved by considering (via discovery or construction) a new intermediate level of representation in addition to the original problem space.…”

“…Similar results have been obtained in research on reversal learning, where a learning agent first learns a set of appropriate behaviors for a given set of situations and then, suddenly, has to show different associations of learned behaviors to the set of situations, in order to reach behavioral goals. While standard RL agents have to be re-trained entirely in such reversal scenarios, biological systems are able to remap hierarchically organized structures of representations apparently created during initial training, and solve subsequent reversal problems much more effectively [19]. There is evidence that biological brains solve such problems by an efficiently structured communication between separable brain modules that implement RL strategies and those that implement knowledge representation [20].…”