Online Selective Kernel-Based Temporal Difference Learning

Abstract

Abstract-In this paper, an online selective kernel-based temporal difference (OSKTD) learning algorithm is proposed to deal with large scale and/or continuous reinforcement learning problems. OSKTD includes two online procedures: online sparsification and parameter updating for the selective kernelbased value function. A new sparsification method (i.e., a kernel distance-based online sparsification method) is proposed based on selective ensemble learning, which is computationally less complex compared with other sparsification methods. With the proposed sparsification method, the sparsified dictionary of samples is constructed online by checking if a sample needs to be added to the sparsified dictionary. In addition, based on local validity, a selective kernel-based value function is proposed to select the best samples from the sample dictionary for the selective kernel-based value function approximator. The parameters of the selective kernel-based value function are iteratively updated by using the temporal difference (TD) learning algorithm combined with the gradient descent technique. The complexity of the online sparsification procedure in the OSKTD algorithm is O(n). In addition, two typical experiments (Maze and Mountain Car) are used to compare with both traditional and up-to-date O(n) algorithms (GTD, GTD2, and TDC using the kernel-based value function), and the results demonstrate the effectiveness of our proposed algorithm. In the Maze problem, OSKTD converges to an optimal policy and converges faster than both traditional and up-to-date algorithms. In the Mountain Car problem, OSKTD converges, requires less computation time compared with other sparsification methods, gets a better local optima than the traditional algorithms, and converges much faster than the upto-date algorithms. In addition, OSKTD can reach a competitive ultimate optima compared with the up-to-date algorithms.Index Terms-Function approximation, online sparsification, reinforcement learning (RL), selective ensemble learning, selective kernel-based value function.

Kernel Temporal Differences for Neural Decoding

“…After 43 trials, we count the number of trials which received a positive reward, and the success rate is averaged over 50 Monte Carlo runs. The performance of the Q -KTD algorithm is compared with Q -learning via time delayed neural net ( Q -TDNN) and the online selective kernel-based temporal difference learning algorithm ( Q -OSKTD) [23] in Figure 9. Note that TDNN is a conventional approach to function approximation and has already been applied to RLBMI experiments for neural decoding [1, 2].…”

“…These methods, known as kernel sparsification methods, can be applied to the KTD algorithm to control the growth of the terms in the function expansion, also known as filter size. Popular examples of kernel sparsification methods are the approximate linear dependence (ALD) [19], Surprise criterion [32], Quantization approach [21], and the kernel distance based method [23]. The main idea of sparsification is to only consider a reduced set of samples, called the dictionary, to represent the function of interest.…”

“…Otherwise, the new input state x ( n ) is absorbed by the closest existing unit. Very similar to the quantization approach, the method presented in [23] applies a distance threshold criterion in the RKHS. The kernel distance based criterion given a state dictionary D ( n − 1) adds a new unit when the new input state x ( n ) satisfies following condition;$\begin{array}{c}\underset{x_i\in D(n-1)}{\min }{‖\varphi \left(x\right(n\left)\right)-\varphi \left(x_i\right)‖}^2>{\mu }_1.\end{array}$For some kernels such as Gaussian, the Quantization method and the kernel distance based criterion can be shown to be equivalent.…”

“…A Quantization approach proposed in [21] has been used in KTD( λ ) [22]. In a similar flavor, the kernel distance based online sparsification method was proposed for a KTD algorithm in [23]. Note that ALD is O ( n 2 ) complexity, whereas quantization and kernel distances are O ( n ).…”

“…The main difference between the quantization approach and the kernel distance is the space where the distances are computed. Quantization approach uses criterion of input space distances whereas kernel distance computes them in the RKHS associated with the kernel [23]. …”