A Competition of Critics in Human Decision-Making

Abstract

Recent experiments and theories of human decision-making suggest positive and negative errors are processed and encoded differently by serotonin and dopamine, with serotonin possibly serving to oppose dopamine and protect against risky decisions. We introduce a temporal difference (TD) model of human decision-making to account for these features. Our model involves two critics, an optimistic learning system and a pessimistic learning system, whose predictions are integrated in time to control how potential decisions compete to be selected. Our model predicts that human decisionmaking can be decomposed along two dimensions: the degree to which the individual is sensitive to (1) risk and (2) uncertainty. In addition, we demonstrate that the model can learn about the mean and standard deviation of rewards, and provide information about reaction time despite not modeling these variables directly. Lastly, we simulate a recent experiment to show how updates of the two learning systems could relate to dopamine and serotonin transients, thereby providing a mathematical formalism to serotonin's hypothesized role as an opponent to dopamine. This new model should be useful for future experiments on human decision-making.

Modeling Human Behavior Part II -- Cognitive approaches and Uncertainty

“…As noted above, biases can be represented in both interpretations of information as well as in the exhibited behavior. This can be demonstrated by Bayesian models of cognitive biases [105], biased probability judgments [139], sensitivity to risk/uncertainty [26], and similar topics relating to representation and interpretation of data/statistics [19,70]. Biases can also be exhibited in how people respond to others or digital avatars (e.g.…”

Having “multiple selves” helps learning agents explore and adapt in complex changing worlds

“…There is also mounting evidence for separated value functions in the human brain (75, 76), and that decision dynamics are best modelled using competing value components (77). Others have proposed that opposed serotonin and dopamine learning systems reflect competition between optimistic and pessimistic behavioral policies (78), and that human behavior can be fit best by assuming it reflects modular reinforcement learning (79, 80). Our work provides a normative basis for these findings, which are all consistent with the idea that different objectives compete for behavioral expression in parallel (81).…”

Integrating Local Learning into the Two-Stage Markov Task to Separate Model-Based from Model-Free Learning

“…By analyzing participants' likelihood of repeating their first-stage choice based on reward outcomes and transition types, researchers aim to measure individual's inclination for MF and MB learning (Daw et al, 2011). However, a purely MF learner can demonstrate the behaviors associated with both MF and MB learning by adjusting the exploration rate (Enkhtaivan et al, 2021). The behavior pattern also varies with how much participants misconstrue the task based on the directions they are given (Feher da Silva and Hare, 2020;Feher da Silva et al, 2023).…”