Improving Robot Motor Learning with Negatively Valenced Reinforcement Signals

Abstract

Both nociception and punishment signals have been used in robotics. However, the potential for using these negatively valenced types of reinforcement learning signals for robot learning has not been exploited in detail yet. Nociceptive signals are primarily used as triggers of preprogrammed action sequences. Punishment signals are typically disembodied, i.e., with no or little relation to the agent-intrinsic limitations, and they are often used to impose behavioral constraints. Here, we provide an alternative approach for nociceptive signals as drivers of learning rather than simple triggers of preprogrammed behavior. Explicitly, we use nociception to expand the state space while we use punishment as a negative reinforcement learning signal. We compare the performance—in terms of task error, the amount of perceived nociception, and length of learned action sequences—of different neural networks imbued with punishment-based reinforcement signals for inverse kinematic learning. We contrast the performance of a version of the neural network that receives nociceptive inputs to that without such a process. Furthermore, we provide evidence that nociception can improve learning—making the algorithm more robust against network initializations—as well as behavioral performance by reducing the task error, perceived nociception, and length of learned action sequences. Moreover, we provide evidence that punishment, at least as typically used within reinforcement learning applications, may be detrimental in all relevant metrics.

Social Reinforcement in Artificial Prelinguistic Development: A Study Using Intrinsically Motivated Exploration Architectures

“…DEVELOPMENT Besides intrinsic motivations, some works have highlighted the relevance of other mechanisms to sensorimotor explo-ration. For instance, proprioception is mentioned in [11,28] and nociception in [29]. Proprioception endows agents with a sense of their own movements.…”

Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks

“…They also o®er the possibility of combining some sort of random search while utilizing the results of previous evaluations. Miikkulainen et al, 25 Navarro-Guerrero et al, 26,27 Real et al, 28 and Xie et al 29 apply evolutionary algorithms to optimize a subset of hyperparameters for neural networks, while Lorenzo et al 30 use a particle swarm optimization algorithm.…”

Neurorobotics—A Thriving Community and a Promising Pathway Toward Intelligent Cognitive Robots

“…I would add (4) Value systems; extracting saliency from the environment and responding appropriately (Friston et al, 1994 ; Krichmar, 2008 ), and (5) Prediction; using past experience to be more successful in the future (Clark, 2013 ). In the area of value systems, models of neuromodulation have been used to simulate value prediction and drive action selection (Sporns and Alexander, 2002 ; Cox and Krichmar, 2009 ; Vargas et al, 2009 ; Krichmar, 2013 ; Navarro-Guerrero et al, 2017 ). Predictive coding strategies using hierarchical Bayesian systems and recurrent neural networks have been used for robots to develop internal models that predict movement of object and of other robots (Park et al, 2012 ; Murata et al, 2017 ).…”