Computational Mechanisms of Osmoregulation: A Reinforcement Learning Model for Sodium Appetite

Abstract

Homeostatic control with oral nutrient intake is a vital complex system involving the orderly interactions between the external and internal senses, behavioral control, reward learning, and decision-making. Sodium appetite is a representative system and has been intensively investigated in animal models of homeostatic systems and oral nutrient intake. However, the system-level mechanisms for regulating sodium intake behavior and homeostatic control remain unclear. In the current study, we attempted to provide a mechanistic understanding of sodium appetite behavior by using a computational model, the homeostatic reinforcement learning model, in which homeostatic behaviors are interpreted as reinforcement learning processes. Through simulation experiments, we confirmed that our homeostatic reinforcement learning model successfully reproduced homeostatic behaviors by regulating sodium appetite. These behaviors include the approach and avoidance behaviors to sodium according to the internal states of individuals. In addition, based on the assumption that the sense of taste is a predictor of changes in the internal state, the homeostatic reinforcement learning model successfully reproduced the previous paradoxical observations of the intragastric infusion test, which cannot be explained by the classical drive reduction theory. Moreover, we extended the homeostatic reinforcement learning model to multimodal data, and successfully reproduced the behavioral tests in which water and sodium appetite were mediated by each other. Finally, through an experimental simulation of chemical manipulation in a specific neural population in the brain stem, we proposed a testable hypothesis for the function of neural circuits involving sodium appetite behavior. The study results support the idea that osmoregulation via sodium appetitive behavior can be understood as a reinforcement learning process, and provide a mechanistic explanation for the underlying neural mechanisms of decision-making related to sodium appetite and homeostatic behavior.

Homeostatic Reinforcement Theory Accounts for Sodium Appetitive State- and Taste-Dependent Dopamine Responding

“…In conclusion, we showed that a homeostatic reinforcement learning theory can account for behaviors motivated by sodium appetite. A recent study also put forth a similar argument [ 62 ]—focusing on a different dataset, they showed that HRRL agents can match two-bottle test behavior for sodium and water choices. Here, we critically extend these ideas to show that the HRRL framework can also quantitatively account for the taste- and state-dependent DA responses, arguing that such DA signaling may in fact be causal for the learning of such appetitive behaviors.…”

Homeostatic Reinforcement Theory Accounts for Sodium Appetitive State- and Taste-Dependent Dopamine Responding

“…Future extensions of the HRRL framework could explore how competing drives interact and influence reward computations when multiple nutrients are available (i.e., hunger vs. thirst, blood osmolality and the drive for water vs. sodium). So far we (Keramati & Gutkin 2014; Keramati & Gutkin 2011) and others (Uchida et al 2022) have considered situations where multiple nutrient sources are independent in their impact on the internal state of the agent and therefore the drive function. Interestingly, we did show that under some conditions multiple sources do interact in their impact on shaping behavior in order to satisfy multiple constraints during behavioral policy formation (ref to Juechems).…”

Homeostatic Control on the Thought: a Comprehensive Explanation of Mind Wandering

“…HRL implements how organisms make decisions to keep their physiological state stable, i.e., the control of behavior by homeostasis, and has potential applications in explaining various behavioral phenomena in organisms (Juechems and Summerfield, 2019; Keramati and Gutkin, 2014; Uchida et al, 2022). This model defines reward as the proximity to a desired value for the internal state (i.e., setpoint).…”

“…A primordial source that drives animals to engage in specific behavior is physiological requirements, such as hunger and thirst. The idea of maintaining a steady physiological state, or homeostasis, has been applied to associative learning, habituation and sensitization, social behavior, and various behavioral phenomena (Eisenstein and Eisenstein, 2006; Juechems and Summerfield, 2019; Keramati and Gutkin, 2014; Lee et al, 2021; Uchida et al, 2022), suggesting that it has a potential to be applicable to human thoughts.…”