Imitation as a model-free process in human reinforcement learning

Abstract

While there is not doubt that social signals affect human reinforcement learning, there is still no consensus about their exact computational implementation. To address this issue, we compared three hypotheses about the algorithmic implementation of imitation in human reinforcement learning. A first hypothesis, decision biasing, postulates that imitation consists in transiently biasing the learner's action selection without affecting her value function. According to the second hypothesis, model-based imitation, the learner infers the demonstrator's value function through inverse reinforcement learning and uses it for action selection. Finally, according to the third hypothesis, value shaping, demonstrator's actions directly affect the learner's value function. We tested these three psychologically plausible hypotheses in two separate experiments (N = 24 and N = 44) featuring a new variant of a social reinforcement learning task, where we manipulated the quantity and the quality of the demonstrator's choices. We show through model comparison that value shaping is favored, which provides a new perspective on how imitation is integrated into human reinforcement learning. Reinforcement Learning | Social Learning | Imitation | Computational cognitive modeling | Decision-making | Meta-learningCorrespondence: anis.najar@ens.fr, stefano.palminteri@ens.fr

Reinforcement learning with human advice: a survey

“…The computational questions covered in this survey extend beyond the boundaries of Artificial Intelligence, as similar research questions regarding the computational implementation of social learning strategies are also raised in the field of Cognitive Neuroscience [10,87,96]. Thus we think this survey can be of interest for both communities.…”

On computational models of theory of mind and the imitative reinforcement learning in spiking neural networks

“…The future direction would be to extend the framework to include more neural populations, representing more complications of the ToM circuits in human brain, and a history of recent events. Given the recent debates on predictive coding and value shaping hypotheses in the context of ToM and social cognition (see 42,43 for instance), it is also of great importance to test if these hypotheses can be explained by an extended ToM-based ImRL framework.…”

On Computational Models of Theory of Mind and the Imitative Reinforcement Learning in Spiking Neural Networks

“…The future direction would be to extend the framework to include more neural populations, representing more complications of the ToM circuits in human brain, and a history of recent events. Given the recent debates on predictive coding and value shaping hypotheses in the context of ToM and social cognition (see 40 and 41 for instance), it is also of great importance to test if these hypotheses can be explained by an extended ToM-based ImRL framework.…”

“…As a result, for each group of components, we respectively have a three-dimensional tensor of shape (n, h, d), where n, h, and d respectively denote the number of features (7 for obstacles, 4 for side margins, 3 for oriented margins, and 1 for shot distances), the height of the frame, and the maximum horizontal distance from the plane/shot. All together, there would be four independent input populations: one for the obstacles of shape (7, 101, 175), one for the side margins of shape (4,25,17), one for the oriented margins of shape (3,101,41), and one for the distance of obstacles from the shot of shape (1,101,21). Note that the lowest bar in the game frame is not considered in ToM-based agent's visual access to reduce the complexity of the network's input (see Figure 5).…”