Zooming for Efficient Model-Free Reinforcement Learning in Metric Spaces

Abstract

Despite the wealth of research into provably efficient reinforcement learning algorithms, most works focus on tabular representation and thus struggle to handle exponentially or infinitely large state-action spaces. In this paper, we consider episodic reinforcement learning with a continuous state-action space which is assumed to be equipped with a natural metric that characterizes the proximity between different states and actions. We propose ZOOMRL, an online algorithm that leverages ideas from continuous bandits to learn an adaptive discretization of the joint space by zooming in more promising and frequently visited regions while carefully balancing the exploitationexploration trade-off. We show that ZOOMRL achieves a worst-case regret O(H) where H is the planning horizon, K is the number of episodes and d is the covering dimension of the space with respect to the metric. Moreover, our algorithm enjoys improved metric-dependent guarantees that reflect the geometry of the underlying space. Finally, we show that our algorithm is robust to small misspecification errors.

Provably adaptive reinforcement learning in metric spaces

“…• Theorem 1 gives a regret bound that depends on the packing numbers of the near-optimal set (Definition 3). This bound should be compared with the "metric-specific" regret guarantee of Sinclair et al (2019) or the "refined regret bound" of Touati et al (2020). Both of these results have the same form as ours with all terms in agreement, but with N pack r (S × A) in the place of N pack r (P Q h,r ).…”

“…Of these, the most related result is that of Sinclair et al (2019) who study the adaptive discretization algorithm and give a worst-case regret analysis, showing that the algorithm has a regret rate of K d+1 d+2 where d is the covering dimension of the metric space. Essentially the same results appear in Touati et al (2020), although the algorithm is slightly different. However, none of these results give sharper instance-dependence guarantees that reflect benign problem structure, as we will obtain.…”

“…Of this line of work, the two most related papers are those of of Jin et al (2018) and Simchowitz and Jamieson (2019). Our results build on the model-free/martingale analysis of Jin et al (2018), which has been used in recent work on RL in metric spaces (Song and Sun, 2019;Sinclair et al, 2019;Touati et al, 2020). We also employ techniques from the gap-dependent analysis of Simchowitz and Jamieson (2019).…”

“…We now state the main assumptions that we adopt in our analysis. These or closely related assumptions are standard in the literature on bandits and reinforcement learning in metric spaces (Song and Sun, 2019;Sinclair et al, 2019;Touati et al, 2020;Slivkins, 2014). (…”

“…We consider episodic RL where the joint state-action space is endowed with a metric and we posit that the optimal Q function is Lipschitz continuous with respect to this metric. This setup has been studied in several recent works establishing worst case regret bounds that scale with the covering dimension of the metric space (Song and Sun, 2019;Sinclair et al, 2019;Touati et al, 2020). While these results are encouraging, the guarantees are overly pessimistic, and intuition from the special case of Lipschitz bandits suggests that much more adaptive guarantees are achievable.…”