‘Proactive’ use of cue-context congruence for building reinforcement learning’s reward function

Abstract

Background: Reinforcement learning is a fundamental form of learning that may be formalized using the Bellman equation. Accordingly an agent determines the state value as the sum of immediate reward and of the discounted value of future states. Thus the value of state is determined by agent related attributes (action set, policy, discount factor) and the agent's knowledge of the environment embodied by the reward function and hidden environmental factors given by the transition probability. The central objective of reinforcement learning is to solve these two functions outside the agent's control either using, or not using a model. Results:In the present paper, using the proactive model of reinforcement learning we offer insight on how the brain creates simplified representations of the environment, and how these representations are organized to support the identification of relevant stimuli and action. Furthermore, we identify neurobiological correlates of our model by suggesting that the reward and policy functions, attributes of the Bellman equitation, are built by the orbitofrontal cortex (OFC) and the anterior cingulate cortex (ACC), respectively. Conclusions:Based on this we propose that the OFC assesses cue-context congruence to activate the most context frame. Furthermore given the bidirectional neuroanatomical link between the OFC and model-free structures, we suggest that model-based input is incorporated into the reward prediction error (RPE) signal, and conversely RPE signal may be used to update the reward-related information of context frames and the policy underlying action selection in the OFC and ACC, respectively. Furthermore clinical implications for cognitive behavioral interventions are discussed.

Alteration of the irisin–brain-derived neurotrophic factor axis contributes to disturbance of mood in COPD patients

“…Expression of FNDC5 was shown to positively correlate with physical activity in several organs (eg, skeletal muscle and brain) as its increased or decreased expression was observed in response to sustained physical training or sedentary lifestyle, respectively 10–12. It is interesting that FNDC5 expression has been shown in the ventral tegmental area (VTA) and hippocampus,13 structures serving model-free and model-based reward-related reinforcement learning processes 14,15. Irisin, the highly conserved fragment of FNDC5, is released into the systemic circulation to exert its most established effect of inducing white adipose tissue browning, activating oxygen consumption, and thermogenesis of fat cells 9,16.…”

Vestibular stimulation and primitive reflex integration may drive multisensory processing: putative principles for a Targeted sensorimotor therapy (TSMT)

“…Spatial memory enables internal simulation and re-representation of the sensory-motor loop's activity in anticipation of future events [20], contributing to a cognitive map possibly used by other processes e.g. model-based reinforcement learning (for an overview see [29,30]). Conversely patients with vestibular dysfunction have been described to suffer from short-term memory loss, concentration, impaired VOR leading to reading disabilities, impaired ability to estimate basic numeric attributes of the environment, like distances and weights, translating into poor arithmetic skills [19].…”

The Alteration of Irisin—Brain-Derived Neurotrophic Factor Axis Parallels Severity of Distress Disorder in Bronchial Asthma Patients

“…Recently, narrowed and rigid contextual learning was proposed as a potential common neurobehavioral mechanism that may underlie distress disorder (Renna et al, 2017 ). Contextual learning is a basic mechanism involved in reinforcement (reward) learning and motivation (Zsuga et al, 2016a , b ) to organize cues and their respective contexts (including rewards) into context frames based on their statistical regularities (Bar, 2007 ). These will serve as the starting point for making forward looking simulations to maximize the sum of future rewards and govern motivated behavior (for an overview see Zsuga et al, 2016b ).…”

“…Contextual learning is a basic mechanism involved in reinforcement (reward) learning and motivation (Zsuga et al, 2016a , b ) to organize cues and their respective contexts (including rewards) into context frames based on their statistical regularities (Bar, 2007 ). These will serve as the starting point for making forward looking simulations to maximize the sum of future rewards and govern motivated behavior (for an overview see Zsuga et al, 2016b ). Characteristic symptoms such as depressive rumination and anxiety have been directly linked to the alteration of contextual learning (Bar, 2009 ) in distress disorder.…”

“…One possible molecular pathway that may link affective constructs (anxiety, rumination) to reinforcement learning is the irisin—brain-derived neurotrophic factor (BDNF) axis, on basis of its potential role in reinforcement learning, given its modulatory effect in structures related to the generation of context frames inherent of reinforcement learning (Zsuga et al, 2016b ). Irisin is a contraction-regulated myokine, a highly conservative protein formed by proteolytic cleavage of fibronectin type III domain-containing protein 5 (FNDC5) (Boström et al, 2012 ).…”