Reward expectations direct learning and drive operant matching in Drosophila

Abstract

Foraging animals must use decision-making strategies that dynamically adapt to the changing availability of rewards in the environment. A wide diversity of animals do this by distributing their choices in proportion to the rewards received from each option, Herrnstein’s operant matching law. Theoretical work suggests an elegant mechanistic explanation for this ubiquitous behavior, as operant matching follows automatically from simple synaptic plasticity rules acting within behaviorally relevant neural circuits. However, no past work has mapped operant matching onto plasticity mechanisms in the brain, leaving the biological relevance of the theory unclear. Here, we discovered operant matching in Drosophila and showed that it requires synaptic plasticity that acts in the mushroom body and incorporates the expectation of reward. We began by developing a dynamic foraging paradigm to measure choices from individual flies as they learn to associate odor cues with probabilistic rewards. We then built a model of the fly mushroom body to explain each fly’s sequential choice behavior using a family of biologically realistic synaptic plasticity rules. As predicted by past theoretical work, we found that synaptic plasticity rules could explain fly matching behavior by incorporating stimulus expectations, reward expectations, or both. However, by optogenetically bypassing the representation of reward expectation, we abolished matching behavior and showed that the plasticity rule must specifically incorporate reward expectations. Altogether, these results reveal the first synapse-level mechanisms of operant matching and provide compelling evidence for the role of reward expectation signals in the fly brain.

Model-based inference of synaptic plasticity rules

“…For this proof-of-principle, our ground-truth network architecture (Figure 3A, top) is inspired by recent studies that have successfully mapped observed behaviors to plasticity rules in the mushroom body (MB) of the fruit fly Drosophila melanogaster (Aso & Rubin, 2016; Rajagopalan et al, 2023; Li et al, 2020; Modi et al, 2020; Davis, 2023). In particular, this work indicates that the difference between received and expected reward information is instrumental in mediating synaptic plasticity (Rajagopalan et al, 2023), and that learning and forgetting happen on comparable timescales (Aso & Rubin, 2016).…”

“…Plasticity occurs exclusively between the input and output layers. We simulate a covariance-based learning rule (Loewenstein & Seung, 2006) known from previous experiments (Rajagopalan et al, 2023). The change in synaptic weight Δ w ij is determined by the presynaptic input x j , and a global reward signal r .…”

“…In extending our model to biological data, we explore its applicability to the decision-making behavior in Drosophila melanogaster that inspired our simulated behavior results. Recent research (Rajagopalan et al, 2023) employed logistic regression to infer learning rules governing synaptic plasticity in the mushroom body, the fly’s neural center for learning and memory. However, logistic regression cannot be used to infer plasticity rules that incorporate recurrent temporal dependencies, such as those that depend on current synaptic weights.…”

“…Curves show 10-trial averaged choice (red) and reward (black) ratio, and horizontal lines the corresponding averages over the 80-trial blocks. (C) Final inferred θ value distribution across 18 flies, comparing models with and without a w ij ( θ 001 ) term and the method from Rajagopalan et al (2023). (D) Left Goodness of fit between fly behavior and model predictions plotted as the percent deviance explained (n = 18 flies).…”

“…(E,F) Same as (C,D), except comparing models that do or don’t incorporated reward expectation. Since these models include weight dependence, they cannot be fit using the method of Rajagopalan et al (2023).…”