Reinforcement Learning Disruptions in Individuals With Depression and Sensitivity to Symptom Change Following Cognitive Behavioral Therapy

Brown, Vanessa M.; Zhu, Lusha; Solway, Alec; Wang, John M.; McCurry, Katherine; King‐Casas, Brooks; Chiu, Pearl H.

doi:10.1001/jamapsychiatry.2021.1844

Cited by 68 publications

(57 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Certainly, the subjective values of states that are taught by the subjective reward functions can vary greatly from their objective values, which is problematic if, for instance, the parameters of the subjective reward function change over time. More generally, given that aspiration levels are an interesting lever for the brain to pull on itself, examining its consequences for psychiatric dysfunction in a range of environments [100,101] is an important direction for future research. It would also be worth exploring whether dysfunctions such as anhedonic depression [102][103][104] partly arise because of problems with subjective rather than objective components of reward sensitivity.…”

Section: Relation To Mood and Anhedoniamentioning

confidence: 99%

The pursuit of happiness: A reinforcement learning perspective on habituation and comparisons

Dubey

Griffiths²,

Dayan³

2022

PLoS Comput Biol

View full text Add to dashboard Cite

In evaluating our choices, we often suffer from two tragic relativities. First, when our lives change for the better, we rapidly habituate to the higher standard of living. Second, we cannot escape comparing ourselves to various relative standards. Habituation and comparisons can be very disruptive to decision-making and happiness, and till date, it remains a puzzle why they have come to be a part of cognition in the first place. Here, we present computational evidence that suggests that these features might play an important role in promoting adaptive behavior. Using the framework of reinforcement learning, we explore the benefit of employing a reward function that, in addition to the reward provided by the underlying task, also depends on prior expectations and relative comparisons. We find that while agents equipped with this reward function are less happy, they learn faster and significantly outperform standard reward-based agents in a wide range of environments. Specifically, we find that relative comparisons speed up learning by providing an exploration incentive to the agents, and prior expectations serve as a useful aid to comparisons, especially in sparsely-rewarded and non-stationary environments. Our simulations also reveal potential drawbacks of this reward function and show that agents perform sub-optimally when comparisons are left unchecked and when there are too many similar options. Together, our results help explain why we are prone to becoming trapped in a cycle of never-ending wants and desires, and may shed light on psychopathologies such as depression, materialism, and overconsumption.

show abstract

Section: Relation To Mood and Anhedoniamentioning

confidence: 99%

The pursuit of happiness: A reinforcement learning perspective on habituation and comparisons

Dubey

Griffiths²,

Dayan³

2022

PLoS Comput Biol

View full text Add to dashboard Cite

show abstract

“…Depression is also treated with cognitive behavioral therapy (CBT), an efficacious psychotherapy theorized to reduce symptoms in part through changing learning ( 114 ). A recent study found that after 12 weeks of CBT, normalization of reward learning rate occurred in patients with depression alongside improved anhedonia and negative affect ( 148 ). Another fMRI study demonstrates that increased neural activity in the ventral striatum, which is known to contain dense regions of dopaminergic terminals, correlates with increased RPE signaling and response to CBT ( 149 ).…”

Section: Depressionmentioning

confidence: 99%

Computational reinforcement learning, reward (and punishment), and dopamine in psychiatric disorders

et al. 2022

View full text Add to dashboard Cite

In the DSM-5, psychiatric diagnoses are made based on self-reported symptoms and clinician-identified signs. Though helpful in choosing potential interventions based on the available regimens, this conceptualization of psychiatric diseases can limit basic science investigation into their underlying causes. The reward prediction error (RPE) hypothesis of dopamine neuron function posits that phasic dopamine signals encode the difference between the rewards a person expects and experiences. The computational framework from which this hypothesis was derived, temporal difference reinforcement learning (TDRL), is largely focused on reward processing rather than punishment learning. Many psychiatric disorders are characterized by aberrant behaviors, expectations, reward processing, and hypothesized dopaminergic signaling, but also characterized by suffering and the inability to change one's behavior despite negative consequences. In this review, we provide an overview of the RPE theory of phasic dopamine neuron activity and review the gains that have been made through the use of computational reinforcement learning theory as a framework for understanding changes in reward processing. The relative dearth of explicit accounts of punishment learning in computational reinforcement learning theory and its application in neuroscience is highlighted as a significant gap in current computational psychiatric research. Four disorders comprise the main focus of this review: two disorders of traditionally hypothesized hyperdopaminergic function, addiction and schizophrenia, followed by two disorders of traditionally hypothesized hypodopaminergic function, depression and post-traumatic stress disorder (PTSD). Insights gained from a reward processing based reinforcement learning framework about underlying dopaminergic mechanisms and the role of punishment learning (when available) are explored in each disorder. Concluding remarks focus on the future directions required to characterize neuropsychiatric disorders with a hypothesized cause of underlying dopaminergic transmission.

show abstract

“…One explanation may be that fear-related symptoms and disorders are more related to these learning differences than distress-related disorders (Clark & Watson, 1991;McTeague & Lang, 2012). Meanwhile, people with anxiety and depression may also learn differently from good versus bad outcomes (Brown et al, 2021;Gagne et al, 2020;Lamba et al, 2020;Wise & Dolan, 2020). The direction of results in individual studies varies; a recent computational meta-analysis suggested that anxiety and depression show greater punishment learning rates (Pike & Robinson, 2022).…”

Section: D Reinforcement Learning Studies Of Anxietymentioning

confidence: 99%

Anxiety as a disorder of uncertainty: Implications for understanding maladaptive anxiety, anxious avoidance, and exposure therapy

Brown¹

2022

Preprint

View full text Add to dashboard Cite

Cognitive-behavioral conceptualizations of anxiety view exaggerated threat expectancies as central to maladaptive anxiety. This view has led to successful treatments, notably exposure therapy, but is not consistent with the empirical literature on learning and choice alterations in anxiety. Empirically, anxiety is better described as a disorder of uncertainty. How disruptions in uncertainty lead to impairing avoidance and are treated with exposure-based methods, however, is unclear. Here, we integrate concepts from neurocomputational learning models with clinical literature on exposure therapy to propose a new framework for understanding maladaptive uncertainty functioning in anxiety, particularly fear-based disorders. Specifically, we propose that anxiety disorders are fundamentally disorders of uncertainty learning and that successful treatments, particularly exposure therapy, work by remediating maladaptive avoidance from dysfunctional explore/exploit decisions in uncertain, potentially aversive situations. This framework reconciles several inconsistencies in the literature and provides a path forward to better understand and treat anxiety.

show abstract

Reinforcement Learning Disruptions in Individuals With Depression and Sensitivity to Symptom Change Following Cognitive Behavioral Therapy

Cited by 68 publications

References 57 publications

The pursuit of happiness: A reinforcement learning perspective on habituation and comparisons

The pursuit of happiness: A reinforcement learning perspective on habituation and comparisons

Computational reinforcement learning, reward (and punishment), and dopamine in psychiatric disorders

Anxiety as a disorder of uncertainty: Implications for understanding maladaptive anxiety, anxious avoidance, and exposure therapy

Contact Info

Product

Resources

About