Optimal Algorithms for Private Online Learning in a Stochastic Environment

Abstract

We consider two variants of private stochastic online learning. The first variant is differentially private stochastic bandits. Previously, (Sajed & Sheffet, 2019) devised the DP Successive Elimination (DP-SE) algorithm that achieves the optimal O 1≤j≤K:∆j >0 log T ∆j + K log T ǫ problem-dependent regret bound, where K is the number of arms, ∆ j is the mean reward gap of arm j, T is the time horizon, and ǫ is the required privacy parameter. However, like other elimination style algorithms, it is not an anytime algorithm. Until now, it was not known whether UCB-based algorithms could achieve this optimal regret bound. We present an anytime, UCB-based algorithm that achieves optimality. Our experiments show that the UCB-based algorithm is competitive with DP-SE. The second variant is the full information version of private stochastic online learning. Specifically, for the problems of decision-theoretic online learning with stochastic rewards, we present the first algorithm that achieves an O log K ∆min + log K ǫ regret bound, where ∆ min is the minimum mean reward gap. The key idea behind our good theoretical guarantees in both settings is the forgetfulness, i.e., decisions are made based on a certain amount of newly obtained observations instead of all the observations obtained from the very beginning.

Differentially Private Reinforcement Learning with Linear Function Approximation

“…Beside the papers mentioned above, there are other related work on differentially private online learning (Guha Thakurta and Smith, 2013;Agarwal and Singh, 2017) and multi-armed bandits (Tossou and Dimitrakakis, 2017;Hu et al, 2021;Sajed and Sheffet, 2019). In the linear bandit setting with contextual information, (Shariff and Sheffet, 2018) shows an impossibility result, i.e., no algorithm can achieve a standard (ε, δ)-DP privacy guarantee while guaranteeing a sublinear regret and thus the relaxed notion of JDP is considered in their paper.…”

Differentially Private Regret Minimization in Episodic Markov Decision Processes

“…2 Garcelon et al (2020) consider stationary transition kernels, Related work. Beside the papers mentioned above, there are other related work on differentially private online learning (Guha Thakurta and Smith 2013; Agarwal and Singh 2017) and multi-armed bandits (Tossou and Dimitrakakis 2017;Hu, Huang, and Mehta 2021;Sajed and Sheffet 2019;Gajane, Urvoy, and Kaufmann 2018;Chen et al 2020). In the RL setting, in addition to Vietri et al (2020); Garcelon et al (2020) that focus on value-iteration based regret minimization algorithms under privacy constraints, Balle, Gomrokchi, and Precup (2016) considers private policy evaluation with linear function approximation.…”

Differentially Private Regret Minimization in Episodic Markov Decision Processes

“…Related work. Beside the papers mentioned above, there are other related work on differentially private online learning (Guha Thakurta and Smith, 2013;Agarwal and Singh, 2017) and multi-armed bandits (Tossou and Dimitrakakis, 2017;Hu et al, 2021;Sajed and Sheffet, 2019;Gajane et al, 2018;Chen et al, 2020). In the RL setting, in addition to Vietri et al (2020); Garcelon et al (2020) that focus on value-iteration based regret minimization algorithms under privacy constraints, Balle et al (2016) considers private policy evaluation with linear function approximation.…”