Exploring Feature Dimensions to Learn a New Policy in an Uninformed Reinforcement Learning Task

Abstract

When making a choice with limited information, we explore new features through trial-and-error to learn how they are related. However, few studies have investigated exploratory behaviour when information is limited. In this study, we address, at both the behavioural and neural level, how, when, and why humans explore new feature dimensions to learn a new policy for choosing a state-space. We designed a novel multi-dimensional reinforcement learning task to encourage participants to explore and learn new features, then used a reinforcement learning algorithm to model policy exploration and learning behaviour. Our results provide the first evidence that, when humans explore new feature dimensions, their values are transferred from the previous policy to the new online (active) policy, as opposed to being learned from scratch. We further demonstrated that exploration may be regulated by the level of cognitive ambiguity, and that this process might be controlled by the frontopolar cortex. This opens up new possibilities of further understanding how humans explore new features in an open-space with limited information.

Holistic Reinforcement Learning: The Role of Structure and Attention

“…This stands in contrast to reinforcement learning models that require definition of an appropriate state space for each task. A simplicity bias is also consistent with findings that suggest that trialand-error learning follows a pattern whereby simpler feature-based state spaces precede more complex object-based spaces [5,58], and explains why classification becomes harder as the number of relevant dimensions grows [59]. The findings outlined in this section illustrate both the importance of structured knowledge in learning and the utility of Bayesian cognitive models for explaining how this knowledge is acquired.…”

Selective particle attention: Rapidly and flexibly selecting features for deep reinforcement learning

“…This is equivalent to a prior over the hypothesis space, favouring hypotheses with relatively few features. Interestingly, this is consistent with findings in neuroscience that people tend to make decisions based on individual features before reasoning about objects that involve more complex combinations of features ( Choung et al, 2017 , Farashahi et al, 2017 ). Similarly, people find it harder to perform classification tasks as the number of relevant dimensions increases ( Shepard, Hovland, & Jenkins, 1961 ).…”

Selective Particle Attention: Visual Feature-Based Attention in Deep Reinforcement Learning

“…This introduces a prior over the hypothesis space, favouring hypotheses with relatively few features. This is consistent with findings that people tend to make decisions based on individual features before reasoning about objects that involve more complex combinations of features (Farashahi et al, 2017;Choung et al, 2017). Similarly, people find it harder to perform classification tasks as the number of relevant dimensions increases (Shepard et al, 1961).…”