Actor-Critic Algorithms with Online Feature Adaptation

Abstract

We develop two new online actor-critic control algorithms with adaptive feature tuning for Markov Decision Processes (MDPs). One of our algorithms is proposed for the long-run average cost objective, while the other works for discounted cost MDPs. Our actor-critic architecture incorporates parameterization both in the policy and the value function. A gradient search in the policy parameters is performed to improve the performance of the actor. The computation of the aforementioned gradient, however, requires an estimate of the value function of the policy corresponding to the current actor parameter. The value function, on the other hand, is approximated using linear function approximation and obtained from the critic. The error in approximation of the value function, however, results in suboptimal policies. In our article, we also update the features by performing a gradient descent on the Grassmannian of features to minimize a mean square Bellman error objective in order to find the best features. The aim is to obtain a good approximation of the value function and thereby ensure convergence of the actor to locally optimal policies. In order to estimate the gradient of the objective in the case of the average cost criterion, we utilize the policy gradient theorem, while in the case of the discounted cost objective, we utilize the simultaneous perturbation stochastic approximation (SPSA) scheme. We prove that our actor-critic algorithms converge to locally optimal policies. Experiments on two different settings show performance improvements resulting from our feature adaptation scheme.

Decentralized Deterministic Multi-Agent Reinforcement Learning

“…Limit Theorem As noted by [6], the fact that the deterministic gradient is a limit case of the stochastic gradient enables the standard machinery of policy gradient, such as compatible-function approximation ( [13]), natural gradients ( [14]), on-line feature adaptation ( [15],) and actor-critic ( [16]) to be used with deterministic policies. We show that it holds in our setting.…”

“…Consider the following on-policy algorithm. The actor step is based on an expression for ∇ θ i J(µ θ ) in terms of ∇ a i Q θ (see Equation (15) in the Appendix). We approximate the action-value function Q θ using a family of functions Qω : S × A → R parameterized by ω, a column vector in R K .…”

Neural Network Compatible Off-Policy Natural Actor-Critic Algorithm

“…Actor-critic algorithms [11], [24] are a popular class of RL algorithms that utilise the policy gradient theorem to compute the optimal policy π θ * . Here, the critic estimates the advantage value function (x) parameters and the actor uses it to improves the policy parameters (θ).…”

“…However, they explicitly compute/approximate a distance metric using the Fisher information matrix (curvature information) giving rise to complex algorithms. This estimation/approximation of the Fisher information matrix can be entirely avoided (see equation (11) of Section III) if the critic employs a linear function approximator with well-defined features known as compatible features [14].…”

Feature selection in deterministic policy gradient

“…Recently, there have been efforts to automate the feature selection in RL. These researchers include performing a gradient descent on manifold of features in the direction of minimising of mean square Bellman error [6, 7], tuning the parameters of feature using a gradient‐based or a cross‐entropy‐based method [8, 9], constricting features with nonparametric techniques [10–13], and expanding the set of features by using TD error [14]. However, these methods do not consider the tradeoff between the approximation error and estimation error.…”