Learning of binocular fixations using anomaly detection with deep reinforcement learning

Abstract

Due to its ability to learn complex behaviors in high-dimensional state-action spaces, deep reinforcement learning algorithms have attracted much interest in the robotics community. For a practical reinforcement learning implementation, reward signals have to be informative in the sense they have to discriminate certain close states and they must not be too noisy. To address the first issue, prior information, e.g. in the form of a geometric model, or human supervision are often assumed. This paper proposes a method to learn binocular fixations without such prior information. Instead, it uses an informative reward requiring little supervised information. The reward computation is based on an anomaly detection mechanism which uses convolutional autoencoders. These detectors estimate in a weakly supervised way an object's pixellic position. This position estimate is affected by noise, which makes the reward signal noisy. We first show that this affects both the learning speed and the resulting policy. Then, we propose a method to partially remove the noise using regression on the detection change given sensor data. The binocular fixation task is learned in a simulated environment on an object training set with various shapes and colors. The learned policy is compared with another one learned with a highly informative and noiseless reward signal. The tests are carried out on the training set and on a test set of new objects. We observe similar performances, showing that the environmentencoding step can replace the prior information.

Learning to Touch Objects Through Stage-Wise Deep Reinforcement Learning

“…First, the robot learns from raw pixels to fixate the object with a two-camera system. For this, we use [23] to learn to fixate an object with weak supervision. At the end of the fixation, the camera system coordinates q camera fix implicitly encode the object position in 3D space.…”

“…Fixating an object means bringing the object at the center of I left and I right by moving the cameras. To learn it, we build on our prior work [23].…”

“…More precisely, we apply a kernel density estimator on the autoencoder reconstruction error images respectively |I left −Î left | and |I right −Î right |. After that, the estimated object pixellic positions, respectively P left o and P right o are at the maximal probabilities as described in [23]. This object detection principle only requires an autoencoder pre-training step without object and the assumption that there is an object in the scene subsequently.…”

“…To build f η , we need to have a database D of input-output pairs (q robot , q camera ) where q camera makes the camera look at the end-effector. To produce such samples, we learn how to fixate the end-effector the same way [23] learns to fixate an object. We use the DDPG algorithm and a reward requiring weak supervision.…”

“…We describe here how we detect the end-effector in the image. Unlike [23] which uses an autoencoder to detect the object as an anomaly, the end-effector image position is computed using the difference in the image before and after pre-defined end-effector finger moves. This end-effector Fig.…”