Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models

Abstract

We develop a probabilistic framework for analysing model-based reinforcement learning in the episodic setting. We then apply it to study finite-time horizon stochastic control problems with linear dynamics but unknown coefficients and convex, but possibly irregular, objective function. Using probabilistic representations, we study regularity of the associated cost functions and establish precise estimates for the performance gap between applying optimal feedback control derived from estimated and true model parameters. We identify conditions under which this performance gap is quadratic, improving the linear performance gap in recent work [X. Guo, A. Hu, and Y. Zhang, arXiv preprint, arXiv:2104.09311, (2021)], which matches the results obtained for stochastic linear-quadratic problems. Next, we propose a phase-based learning algorithm for which we show how to optimise exploration-exploitation trade-off and achieve sublinear regrets in high probability and expectation. When assumptions needed for the quadratic performance gap hold, the algorithm achieves an order O( √ N ln N ) high probability regret, in the general case, and an order O((ln N ) 2 ) expected regret, in self-exploration case, over N episodes, matching the best possible results from the literature. The analysis requires novel concentration inequalities for correlated continuous-time observations, which we derive.

Statistical Learning with Sublinear Regret of Propagator Models

“…Our main results which were outlined above significantly extend the work on reinforcement learning for continuous-time parametric models which were studied by [4,27,42,17,18,43] among others. We outline our main contributions that correspond to each part of the learning algorithm.…”

“…The main ingredient in the exploration phase is to estimate the kernel function G in (1.1), in non-parametric manner. This stands in contrast to existing theoretical works on parametric model-based reinforcement learning problems [4,27,42,17,18,43]. Moreover, in our setting G is the driver of the transient price impact which is an unobserved state variable of the control problem.…”

“…Lipschitz stability of infinite-dimensional optimal control: In [4,27,42], Lipschitz stability of optimal controls for stochastic control problems was derived, by showing that the optimiser is continuously differentiable with respect to finitedimensional model parameters, hence establishing Lipschitz continuity. The Lipschitz stability of controls is crucial for quantifying the precise performance gap between controls derived from estimated and true models, and for characterizing the regret order of learning algorithms.…”

“…In Theorem 2.18 and Corollary 2.20 we derive a bound on the regret after N trading episodes, which is of order N 3/4 for a regular kernel and of order N 3−2α 4−4α for a singular kernel. These sublinear regret bounds underperform the square-root (or logarithmic) regret for reinforcement learning problems with finite-dimensional parametric models (see e.g., [4,27,42,17,18,43]), due to the present infinite-dimensional non-parametric kernel estimation (see Remark 2.19).…”

Linear convergence of a policy gradient method for finite horizon continuous time stochastic control problems

“…By exploiting the boundedness of ∂ x f and ∂ x g, we establish an a-priori bound of the adjoint processes, and subsequently prove the iterative scheme (1.7) generates Lipschitz continuous policies (φ m ) m∈N 0 (see Proposition 3.7). If the drift and diffusion coefficients are affine in x, then (2.1) and (2.5) can be relaxed to quadratically growing functions, which include as special cases the linear-convex control problems studied in [12,36].…”