Reinforcement learning analysis for a minimum time balance problem

Abstract

Reinforcement learning was developed to solve complex learning control problems, where only a minimal amount of a priori knowledge exists about the system dynamics. It has also been used as a model of cognitive learning in humans and applied to systems, such as pole balancing and humanoid robots, to study embodied cognition. However, closed-form analysis of the value function learning based on a higher-order unstable test problem dynamics has been rarely considered. In this paper, firstly, a second-order, unstable balance test problem is used to investigate issues associated with the value function parameter convergence and rate of convergence. In particular, the convergence of the minimum time value function is analysed, where the minimum time optimal control policy is assumed known. It is shown that the temporal difference error introduces a null space associated with the experiment termination basis function during the simulation. As this effect occurs due to termination or any kind of switching in control signal, this null space appears in temporal differences (TD) error for more general higher-order systems. Secondly, the rate of parameter convergence is analysed and it is shown that residual gradient algorithm converges faster than TD(0) for this particular test problem. Thirdly, impact of the finite horizon on both the value function and control policy learning has been analysed in case of unknown control policy and added random exploration noise.

A variable resolution feedback improving the performances of object detection and recognition

“…However, the proposed method performs higher accuracy and precision than the method with fixed resolution 640 × 480. Furthermore, the time consumption per image of the proposed method is less than that of conventional methods (Figure 7(b)), which is 44.1% of the method with fixed resolution 640 × 480 and 28.1% of the method with fixed resolution 1280 × 960. GBVS model and dense SIFT algorithm cost the majority of the whole time, and the higher the resolution, the more time they cost.…”

“…Therefore, such situations result in low speed of image processing. Tutsoy and Brown 7 prove that a learning/classification algorithm with optimized control policy can precisely produce the same result as the analytical solution with the amount of data reduced. Besides, processing images with high resolution for object detection and recognition is not economical, because interesting objects do not always show up in FOV.…”

Incremental and Parallel Machine Learning Algorithms With Automated Learning Rate Adjustments

“…A concrete test problem with a closed-form solution is a good way to evaluate the performance of algorithms in detail (Tutsoy and Brown, 2016a , b ). Here, we used the existing and proposed algorithms to solve a simple problem.…”

Alera

“…A few works [46,47] have also explored ideas from Probably Approximately Correct (PAC) learning framework to reuse previously discarded experiences intelligently to accelerate the value function learning. Recently, in [40], the authors have analyzed the rate of parameter convergence for RL algorithms in presence of unstable system dynamics and random exploration noise, thus showing the significant potential of accelerating the learning process.…”