Computational and neural mechanisms underlying the influence of action affordances on value learning

Abstract

When encountering a novel situation, an intelligent agent needs to find out which actions are most beneficial for interacting with that environment. However, the range of possible actions that could be selected is virtually unlimited, making the problem of determining which subset of actions should be drawn from to begin exploration extremely challenging. One purported mechanism for narrowing down the scope of possible actions is the concept of action affordance. Here, we delve into the neuro-computational mechanisms accounting for how action affordance shapes decision-making and action-selection by utilizing a novel task alongside computational modeling of behavioral and fMRI data collected in humans. Our findings indicate that rather than simply exerting an initial or persistent bias on value-driven choices, action affordance is better conceived of as an independent system that concurrently guides action-selection alongside value-based decision-making. These two systems engage in a competitive process to determine final action selection, governed by a dynamic meta controller. We find that the posterior parietal cortex plays a central role in integrating the predictions from these two controllers of what action to select, so that the action-selection process dynamically takes into account both the expected value and appropriateness of particular actions for a given scenario.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

“…Although mostly overlooked as part of models of value-based learning, constant bias has occasionally been reported-with and without laterality [12,[14][15][16][17]21,91,[94][95][96][97][98][99] as well as between acting and not acting for a go/no-go task [100][101][102][103][104][105]. Even decision making that is not defined by learning-whether value-based [106] or perceptual [80,86,[88][89][90]92,[107][108][109][110][111][112]-can be affected by such stimulus-independent biases with a less obvious role for bias than would be assumed for skillful action-based decision making where physical aspects of action per se have explicit relevance [113].…”

“…Even low-level motor biases, which if disregarding their benefits in lower internal cost might otherwise be considered a disadvantage of embodiment, may also not be so disruptive as part of a tradeoff for which an embodied RL policy has greater potential for robustness in learning per se. Indeed, embodied RL for concrete actions can achieve greater fluency than disembodied RL for symbolic choices abstracted away from motor output [95,99,202]. Benefits of embodied learning may be facilitated by lesser working-memory demands and lesser overall demands from the topology of the action space as a cognitive map [370][371][372][373] more amenable to spatial and embodied representations in the neural circuitry of the basal ganglia and cortex [8,21,22,[374][375][376][377][378].…”

“…For example, the environment here was structured to be conducive to an alternation bias via second-order perseveration. Adaptive bias has been suggested for actions, effectors, or spatial locations in experimental paradigms delivering rewards asymmetrically with distributions that are congruent or incongruent with respect to particular biases [99,106,111,210,212,213,300,301,333,379]. Adaptive control with the heuristics of a mixture policy would entail flexible leveraging or suppressing of action bias and hysteresis to strike a balance among various tradeoffs of bias and variance, speed and accuracy, energy and effort, benefit and cost, reward and compressibility, expertise and efficiency, or exploration and exploitation.…”

“…As with the mixture-of-experts (MoE) architecture per se (which has also proven effective for sparse scaling of a deep neural network), the scope of this consilient theory can be extended to systems of varying levels of expertise as well as nonexpert controllers and their numerous choice and action biases (cf. [12,21,81,86,[88][89][90][91][92][94][95][96]98,99,143,144]). Benefiting from distributed control of decisions and actions across diverse levels of representation in the networks of the nervous system [227][228][229], a mixture of experts and nonexperts can dynamically mediate distinct subpolicies with the metacontrol of a manager for arbitration over the gated ensemble of modular learning and nonlearning processes.…”