No learning where to go without first knowing where you're coming from: action discovery is trajectory, not endpoint based

Abstract

Intrinsic motivations drive an agent to explore, providing essential data for linking behaviors with novel outcomes and so laying the foundation for future flexible action. We present experiments using a new behavioral task which allows us to interrogate the connection between exploration and action learning. Human participants used a joystick to search repeatedly for a target location, only receiving feedback on successful discovery. Feedback delay was manipulated, as was the starting position. Experiment 1 employed stable starting positions, so the task could be learnt with respect to a target location or a target trajectory. Participants were able to learn the correct movement under all delay conditions. Experiment 2 used a variable starting location, so the correct movement could only be learnt in terms of target location. Participants displayed little to no learning in this experiment. These results suggest that movements on this scale are stored as trajectories rather than in terms of target location. Overall the experiments demonstrate the potential of this task for uncovering the native representational substrates of action learning.

Intrinsic motivations and open-ended development in animals, humans, and robots: an overview

“…The model, also tested with the simulated humanoid robot iCub, shows how these modulatory mechanisms can play important adaptive functions for the control of overt attention, manipulation, and goal-directed processes. Thirkettle et al ( 2013 ) introduce the novel “Joystick experimental paradigm” developed to study intrinsically and extrinsically driven acquisition of actions. The authors demonstrate the function and effectiveness of this paradigm by presenting behavioral experiments grounded in the neuroscientific literature and concerning the acquisition of non-trivial motor actions.…”

Emergent structured transition from variation to repetition in a biologically-plausible model of learning in basal ganglia

“…End-points that correspond to a target location elicit a reinforcement signal, and, importantly, reinforcement is not contingent on movement trajectory. The model task is inspired by behavioral counterparts we have used to study action discovery in which participants manipulate a joystick to find an invisible target area in the workspace (Stafford et al, 2012 , 2013 ; Thirkettle et al, 2013a , b ). While there may be “gestural” aspects of action in the behavioral task, in the model we ignore these and focus only on the spatial location of movement end-point.…”