Stress, genotype and norepinephrine in the prediction of mouse behavior using reinforcement learning

Abstract

Individual behavioral performance during learning is known to be affected by modulatory factors, such as stress and motivation, and by genetic predispositions that influence sensitivity to these factors. Despite numerous studies, no integrative framework is available that could predict how a given animal would perform a certain learning task in a realistic situation. We found that a simple reinforcement learning model can predict mouse behavior in a hole-box conditioning task if model metaparameters are dynamically controlled on the basis of the mouse's genotype and phenotype, stress conditions, recent performance feedback and pharmacological manipulations of adrenergic alpha-2 receptors. We find that stress and motivation affect behavioral performance by altering the exploration-exploitation balance in a genotype-dependent manner. Our results also provide computational insights into how an inverted U-shape relation between stress/arousal/norepinephrine levels and behavioral performance could be explained through changes in task performance accuracy and future reward discounting.Animal behavior is guided by rewards that can be received in different situations and by modulatory factors such as stress and motivation. Acute stress can have positive or negative effects on learning and memory that depend on stressor properties (timing, duration and relation with the task) and on the predispositions of stressed individuals [1][2][3] . These effects are thought to be mediated through the modulation of synaptic plasticity by stress hormones and neuromodulators, such as glucocorticoids and norepinephrine [4][5][6][7] . However, their role in high-level processes such as learning, action selection and future reward discounting is not well understood. In addition to stress, genotype 8 , affective traits 9 , motivation 10 and recent performance feedback 11 also influence individual performance, but it may be difficult and inefficient to explicitly model each factor to accurately predict animal behavior.A number of models have related neuromodulatory systems to cognitive processes and to statistical quantities characterizing the environment 12,13 . Although such models provide insights into potential mechanisms, alone they are often unable to accurately predict animal behavior in a realistic situation as a result of the diversity of the modulatory factors affecting it. Here, we propose a method that can, in principle, quantify the influence of arbitrary modulatory factors on behavior as control parameters of a general behavioral model and that is exemplified here for the case of stress and genetic strain in mice.In modeling reward-based behavioral learning, approaches based on the theory of reinforcement learning 14 have been the most successful. They have been applied to explain experimental data in animal conditioning 10 , human decision-making 15 and addiction 16 . However, the modulatory role of stress and affective traits has not yet been considered. In reinforcement learning, modeled animals occupy different states cor...

Stress at Learning Facilitates Memory Formation by Regulating AMPA Receptor Trafficking Through a Glucocorticoid Action

“…The question thus arises as to which mechanisms will be involved when animals are trained under high stress that leads to learning impairment. Importantly, using a computational model, we have recently proposed that the facilitating actions of stress alter the trade between animals' exploration vs explotaition of the current knowledge (translated in changes in task performance accuracy), while high stress levels might impair learning by increasing animals' impulsivity (manifested as changes in future reward discounting) (Luksys et al, 2009). …”

Correction: BAIAP2 Is Related to Emotional Modulation of Human Memory Strength

“…However, the merits of model-based studies should not be judged in isolation, but compared to the alternatives, such as raw behavioral variables or their principal components, which often lack specificity, interpretability and may not generalize to different populations, tasks, and phenotypes. Even very simple models are useful if they are supported by empirical evidence such as neural or genetic correlates, which can enable prediction of individual cognitive parameters based on various modulatory factors (as was shown in the model-based study of mouse behavior [21]). Such predictive capabilities will ultimately help design efficient, simulation-based means to test cognitive and pharmacological manipulations that could be useful for improving cognitive abilities and treating neuropsychiatric disorders.…”

“…Candidategene studies related genetic polymorphisms in dopaminergic genes to specific reinforcement learning parameters [19,20]. Modelbased analysis was also used to investigate how stress, motivation, and noradrenergic manipulations influence different reinforcement learning parameters [21], leading to a novel computational interpretation of the inverted-U-shape relationship between stress and behavioral performance. Model-based analyses, however, have not yet been widely used outside the realm of reinforcement learning and decision-making, nor were they applied to GWAS.…”

Computational dissection of human episodic memory reveals mental process-specific genetic profiles

“…To evaluate how well the model with a particular set of parameters fits individual behavioral performance, we used the following goodness-of-fit function (20,21,68):…”

“…Because some of the mental processes involved in memory are not readily amenable to direct observation, computational modeling can be used to make inferences about them (13,14). Modelbased analyses provided insights into neurocomputational mechanisms of reward-based learning and decision making (15)(16)(17), related model parameters such as the learning rate to genetic polymorphisms (18,19), and provided a computational explanation for the inverted-U-shaped relation between stress intensity and behavioral performance (20). As a relatively recent development, the model-based analysis approach has largely been missing from studies of human episodic memory and genome-wide association studies (GWAS).…”