Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

Abstract

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single rewardprediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversallearning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With highresolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

Decision heuristics in contexts integrating action selection and execution

“…The first question relates to robustness over time. Given the tendency for learning-related configurations in the human brain to vary more across rather than within individuals 50 , we employed a between-groups analytic approach inspired by an increasing body of work that uses behavioural profiles to cluster groups of individuals to increase robustness and reliability of hypothesis-specific brain activity 34,51,52 . While the present DDM parsimoniously distinguished human participants based on strategy-specific modulations of drift rate, parameters were necessarily static.…”

Decision heuristics in contexts exploiting intrinsic skill

“…An additional key outstanding question relates to the robustness of heuristic adoption over time. Given the tendency for learning-related configurations in the human brain to vary more across rather than within individuals 40 , we employed a between-groups analytic approach inspired by an increasing body of work that uses behavioural profiles to cluster groups of individuals to increase robustness and reliability of hypothesis-specific brain activity 24, 41, 53 . While the present DDM parsimoniously distinguished human participants based on strategy-specific modulations of drift-rate, parameters were necessarily static.…”

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

“…The RL framework has appreciable predictive validity [4,5] when accounting for human choices and learning behavior in a variety of settings [6][7][8]-let alone the power of extensions of RL [9][10][11][12]. However, such models sometimes fail to account well for an individual's behavior even in a relatively simple task that should be amenable to RL in principle [13].…”

“…Hysteresis is bidirectional as well and adds dynamics in the form of either repetition or alternation of previous actions, which may also manifest for a horizon beyond just the most recent action [17][18][19][20]. Despite at least some precedent for either action bias or action hysteresis (more so the latter), the combination of both bias and hysteresis has even less precedent for RL [12,21].…”