Flexible Working Memory Through Selective Gating and Attentional Tagging

Abstract

Working memory is essential: it serves to guide intelligent behavior of humans and nonhuman primates when task-relevant stimuli are no longer present to the senses. Moreover, complex tasks often require that multiple working memory representations can be flexibly and independently maintained, prioritized, and updated according to changing task demands. Thus far, neural network models of working memory have been unable to offer an integrative account of how such control mechanisms can be acquired in a biologically plausible manner. Here, we present WorkMATe, a neural network architecture that models cognitive control over working memory content and learns the appropriate control operations needed to solve complex working memory tasks. Key components of the model include a gated memory circuit that is controlled by internal actions, encoding sensory information through untrained connections, and a neural circuit that matches sensory inputs to memory content. The network is trained by means of a biologically plausible reinforcement learning rule that relies on attentional feedback and reward prediction errors to guide synaptic updates. We demonstrate that the model successfully acquires policies to solve classical working memory tasks, such as delayed recognition and delayed pro-saccade/anti-saccade tasks. In addition, the model solves much more complex tasks, including the hierarchical 12-AX task or the ABAB ordered recognition task, both of which demand an agent to independently store and updated multiple items separately in memory. Furthermore, the control strategies that the model acquires for these tasks subsequently generalize to new task contexts with novel stimuli, thus bringing symbolic production rule qualities to a neural network architecture. As such, WorkMATe provides a new solution for the neural implementation of flexible memory control.

Priority-based transformations of stimulus representation in visual working memory

“…A hallmark of WM in the real world is the ability to flexibly respond to unpredictable changes in environmental exigencies. Thus, an important future goal will be to extend the present work to a network with separate modules with different connectivity patterns and governed by different learning rules (e.g., Kruijne et al, 2020;O'Reilly & Frank, 2006).…”

Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making

“…To quantify expectations, we assume that participants build an internal model of the environment (‘world-model’), i.e., we hypothesize that participants estimate the probability p ( t ) ( s t +1 | s t , a t ) of a transition from a given state s t to another state s t +1 when performing action a t . More precisely, we assume that the world-model counts transitions from state s to s ′ under action a using either a leaky [ 23 , 49 , 50 ] or a surprise-modulated [ 28 , 29 , 51 ] counting procedure, described by the pseudo-count . The conditional probability is then where ϵ is a parameter corresponding to a prior in the Bayesian framework, 11 is the total number of states in the environment, and is the pseudo-count of taking action a t at state s t (see Methods and S1 Text ).…”

“…Therefore, we consider the surprise of such a transition to be a decreasing function of p ( t ) ( s t +1 | s t , a t ). More precisely, we use a recent measure of surprise motivated by a Bayesian framework for learning in volatile environments, called the ‘Bayes Factor’ surprise [ 29 ]. The Bayes Factor surprise of the transition from state s t to state s t +1 after taking action a t is where p ( t ) ( s t +1 | s t , a t ) is the conditional probability of observing state s t +1 at time t + 1 derived from the present world-model.…”

“…Our key findings can be summarized in three points: (i) We find that novelty-seeking explains participants’ exploratory behavior better than alternative exploration strategies such as seeking surprise or uncertainty [ 42 , 43 ]; (ii) we observe that participants use their world-model only rarely for action planning and mainly to extract moments of surprise; and importantly, (iii) we show that surprise calculated by the world-model does not only modulate the learning of the world-model [ 24 – 26 , 29 ] but also the learning of model-free action-values. In particular, we show that such a modulation is necessary to explain participants’ adaptive behavior.…”

A Novel Poly-N-Epoxy Propyl Carbazole Based Memory Device

“…[ 1 , 2 , 3 , 4 ]. As a result, these devices are an excellent choice for light-emitting diodes, solar cells, low-cost RFID, sensors, and many other electronic devices [ 5 , 6 , 7 , 8 ]. For a number of applications, such as mobile phones, smart watches, and other electronic devices, low-cost and flexible memory devices are essential [ 9 , 10 ].…”