Reward expectations direct learning and drive operant matching inDrosophila

Abstract

Foraging animals must use decision-making strategies that dynamically account for uncertainty in the world. To cope with this uncertainty, animals have developed strikingly convergent strategies that use information about multiple past choices and reward to learn representations of the current state of the world. However, the underlying learning rules that drive the required learning have remained unclear. Here, working in the relatively simple nervous system of Drosophila, we combine behavioral measurements, mathematical modeling, and neural circuit perturbations to show that dynamic foraging depends on a learning rule incorporating reward expectation. Using a novel olfactory dynamic foraging task, we characterize the behavioral strategies used by individual flies when faced with unpredictable rewards and show, for the first time, that they perform operant matching. We build on past theoretical work and demonstrate that this strategy requires the existence of a covariance-based learning rule in the mushroom body - a hub for learning in the fly. In particular, the behavioral consequences of optogenetic perturbation experiments suggest that this learning rule incorporates reward expectation. Our results identify a key element of the algorithm underlying dynamic foraging in flies and suggest a comprehensive mechanism that could be fundamental to these behaviors across species.

Hacking brain development to test models of sensory coding

“…With PAM-DAN reinforcement, sham-treated control animals chose the rewarded odor in 59% of trials on average, compared to 51% prior to training (Figures 6D, S6B) (Rajagopalan et al, 2022). Hydroxyurea-treated animals that retained all four Kenyon cell clones in both hemispheres ("[4,4]" set) learned similarly to controls, while animals with all 8 clones lost ("[0,0]") failed to learn (Figures 6D, S6B).…”

“…Again, the learning scores of individual flies were assessed and their mushroom body anatomy evaluated by post-hoc staining. In this set of experiments, we provided optogenetic reward using the Gr64f population of sugar-sensing gustatory neurons, which produced much better learning scores in control animals than PAM-DAN reinforcement (Figure 7A); an explanation for this difference is discussed in (Rajagopalan et al, 2022). The proportion of trials with correct odor choices was quantified in naïve and rewarded trials (Figure 7B, C).…”

“…Single fly behavior experiments were performed in a novel olfactory Y-arena. A detailed description of the apparatus is provided in (Rajagopalan et al, 2022). Briefly, the Y chamber consists of two layers of white opaque plastic.…”

A vast space of compact strategies for highly efficient decisions

“…To illustrate this space, we focus on a well-studied task in which an animal, or agent, gathers rewards from two ports whose reward probabilities change dynamically over time according to a hidden world state [11, 12, 13, 14, 15, 16]. To improve performance on this task, the agent can leverage knowledge about the volatility of the environment and the probability of rewards at each port in order to update an internal estimate of the world state based on the outcomes of its past actions.…”

“…Recent studies have suggested that animals leverage such inferences to guide decisions in the face of uncertainty [18, 19, 20]. In the absence of such inferences, other studies have shown how animals could directly estimate the value of individual ports, and use these estimates to guide behavior [21, 14, 15]. These studies focus on a small number of candidate strategies that can optimally guide behavior based on precise internal estimates of world variables.…”