Online learning in Markov decision processes with arbitrarily changing rewards and transitions

Yu, Jia Yuan; Mannor, Shie

doi:10.1109/gamenets.2009.5137416

Cited by 20 publications

(13 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also mention here that Yu and Mannor [20,21] considered the related problem of online learning in MDPs where the transition probabilities may also change arbitrarily after each transition. This problem is significantly more difficult than the case where only the reward function is allowed to change.…”

Section: Introductionmentioning

confidence: 99%

Online Markov Decision Processes Under Bandit Feedback

Neu

György

Szepesvári

et al. 2014

IEEE Trans. Automat. Contr.

137

110

View full text Add to dashboard Cite

International audienceWe consider online learning in finite stochastic Markovian environments where in each time step a new reward function is chosen by an oblivious adversary. The goal of the learning agent is to compete with the best stationary policy in hindsight in terms of the total reward received. Specifically, in each time step the agent observes the current state and the reward associated with the last transition, however, the agent does not observe the rewards associated with other state-action pairs. The agent is assumed to know the transition probabilities. The state of the art result for this setting is an algorithm with an expected regret of O(T^2/3 ln T). In this paper, assuming that stationary policies mix uniformly fast, we show that after T time steps, the expected regret of this algorithm (more precisely, a slightly modified version thereof) is O(T^1/2 ln T), giving the first rigorously proven, essentially tight regret bound for the problem

show abstract

Section: Introductionmentioning

confidence: 99%

Online Markov Decision Processes Under Bandit Feedback

Neu

György

Szepesvári

et al. 2014

IEEE Trans. Automat. Contr.

137

110

View full text Add to dashboard Cite

show abstract

“…I n particular, at each time step n ∈ {1, 2, ...}, the server picks a caching action a n from the action space A given the current state of the system g n ∈ G, which is the state space. Given the current action and state pair (g n , a n ), the server moves to some state g with the probability of Pr(g |g n , a n ) and receives a reward r n (g n , a n ) [28]. In the following, the action space, states, transition probabilities, and reward function are defined.…”

Section: System Modelmentioning

confidence: 99%

Contextual Learning for Content Caching With Unknown Time-Varying Popularity Profiles via Incremental Clustering

Liu

Derakhshani

Lambotharan

2021

IEEE Trans. Commun.

View full text Add to dashboard Cite

With the rapid development of social networks and high-quality video sharing services, the demand for delivering large quantity and high quality contents under stringent endto-end delay requirement is increasing. To meet this demand, we study the content caching problem modelled as a Markov decision process in the network edge server when the popularity profiles are unknown and time-varying. In order to adapt to the changing trends of content popularity, a context-aware popularity learning algorithm is proposed. We prove that the learning error of this scheme is sublinear in the number of requests. In light of the learned popularities, a reinforcement learningbased caching scheme is designed on top of the state-actionreward-state-action algorithm with a function approximation. A reactive caching algorithm is also proposed to reduce the complexity. The time complexities of both the caching schemes are studied to demonstrate their feasibility in real time systems and a theoretical analysis is performed to prove that the cache hit rate of the reactive caching algorithm asymptotically converges to the optimal cache hit rate. Finally the simulations are presented to demonstrate the superiority of the proposed algorithms.

show abstract

“…Non-stationarity has attracted interest in the multi-armed bandit (MAB) and RL literature. Prior RL works typically deal with general non-stationary environments, e.g., (Goldberg and Matarić 2003;da Silva et al 2006;Yu and Mannor 2009;Al-Shedivat et al 2018). A relevant approach for efficient policy adaptation is to meta-learn the changes in the reward and update the policy accordingly (Al-Shedivat et al 2018).…”

Section: Non-stationary Rewardsmentioning

confidence: 99%

Multi-objective Neural Architecture Search via Non-stationary Policy Gradient

Chen,

Zhou,

Trimponias

et al. 2020

Preprint

View full text Add to dashboard Cite

Multi-objective Neural Architecture Search (NAS) aims to discover novel architectures in the presence of multiple conflicting objectives. Despite recent progress, the problem of approximating the full Pareto front accurately and efficiently remains challenging. In this work, we explore the novel reinforcement learning (RL) based paradigm of non-stationary policy gradient (NPG). NPG utilizes a non-stationary reward function, and encourages a continuous adaptation of the policy to capture the entire Pareto front efficiently. We introduce two novel reward functions with elements from the dominant paradigms of scalarization and evolution. To handle non-stationarity, we propose a new exploration scheme using cosine temperature decay with warm restarts. For fast and accurate architecture evaluation, we introduce a novel pretrained shared model that we continuously fine-tune throughout training. Our extensive experimental study with various datasets shows that our framework can approximate the full Pareto front well at fast speeds. Moreover, our discovered cells can achieve supreme predictive performance compared to other multi-objective NAS methods, and other singleobjective NAS methods at similar network sizes. Our work demonstrates the potential of NPG as a simple, efficient, and effective paradigm for multi-objective NAS.

show abstract

Online learning in Markov decision processes with arbitrarily changing rewards and transitions

Cited by 20 publications

References 23 publications

Online Markov Decision Processes Under Bandit Feedback

Online Markov Decision Processes Under Bandit Feedback

Contextual Learning for Content Caching With Unknown Time-Varying Popularity Profiles via Incremental Clustering

Multi-objective Neural Architecture Search via Non-stationary Policy Gradient

Contact Info

Product

Resources

About