A Viscosity Approach to Stochastic Differential Games of Control and Stopping Involving Impulsive Control

Mguni, David

doi:10.48550/arxiv.1803.11432

Cited by 6 publications

(7 citation statements)

References 24 publications

(43 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In continuous-time optimal control theory [24], problems in which the agent faces a cost for each action are tackled with a form a policy known as impulse control [22,19,2]. In impulse control frameworks, the dynamics of the system are modified through a sequence of discrete actions or bursts chosen at times that the agent chooses to apply the control policy.…”

Section: Related Workmentioning

confidence: 99%

“…In this paper, we tackle this problem by developing an RL framework for finding both an optimal criterion to determine whether or not to execute actions as well as learning optimal actions. A key component of our framework is a novel combination of RL with a form of policy known as impulse control [22,19]. This enables the agent to determine the appropriate points to perform an action as well as the optimal action itself.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints

Mguni¹,

Sootla²,

Ziomek³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Many real-world settings involve costs for performing actions; transaction costs in financial systems and fuel costs being common examples. In these settings, performing actions at each time step quickly accumulates costs leading to vastly suboptimal outcomes. Additionally, repeatedly acting produces wear and tear and ultimately, damage. Determining when to act is crucial for achieving successful outcomes and yet, the challenge of efficiently learning to behave optimally when actions incur minimally bounded costs remains unresolved. In this paper, we introduce a reinforcement learning (RL) framework named Learnable Impulse Control Reinforcement Algorithm (LICRA), for learning to optimally select both when to act and which actions to take when actions incur costs. At the core of LICRA is a nested structure that combines RL and a form of policy known as impulse control which learns to maximise objectives when actions incur costs. We prove that LICRA, which seamlessly adopts any RL method, converges to policies that optimally select when to perform actions and their optimal magnitudes. We then augment LICRA to handle problems in which the agent can perform at most k < ∞ actions and more generally, faces a budget constraint. We show LICRA learns the optimal value function and ensures budget constraints are satisfied almost surely. We demonstrate empirically LICRA's superior performance against benchmark RL methods in OpenAI gym's Lunar Lander and in Highway environments and a variant of the Merton portfolio problem within finance.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints

Mguni¹,

Sootla²,

Ziomek³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Generator adaptively guides the agents' exploration and behaviour towards coordination and maximal joint performance. A pivotal feature of LIGS is the novel combination of RL and switching controls Bayraktar & Egami (2010); Mguni (2018) which enables it to determine the best set of states to learn to add intrinsic rewards while disregarding less useful states. This enables Generator to quickly learn how to set intrinsic rewards that guide the agents during their learning process.…”

Section: Introductionmentioning

confidence: 99%

LIGS: Learnable Intrinsic-Reward Generation Selection for Multi-Agent Learning

Mguni¹,

Jafferjee²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Efficient exploration is important for reinforcement learners (RL) to achieve high rewards. In multi-agent systems, coordinated exploration and behaviour is critical for agents to jointly achieve optimal outcomes. In this paper, we introduce a new general framework for improving coordination and performance of multi-agent reinforcement learners (MARL). Our framework, named Learnable Intrinsic-Reward Generation Selection algorithm (LIGS) introduces an adaptive learner, Generator that observes the agents and learns to construct intrinsic rewards online that coordinate the agents' joint exploration and joint behaviour. Using a novel combination of reinforcement learning (RL) and switching controls, LIGS determines the best states to learn to add intrinsic rewards which leads to a highly efficient learning process. LIGS can subdivide complex tasks making them easier to solve and enables systems of RL agents to quickly solve environments with sparse rewards. LIGS can seamlessly adopt existing multi-agent RL algorithms and our theory shows that it ensures convergence to joint policies that deliver higher system performance. We demonstrate the superior performance of the LIGS framework in challenging tasks in Foraging and StarCraft II. To aid coordinated learning, algorithms such as QMIX Rashid et al. (2018), MAVEN Mahajan et al. (2019) and COMA Foerster et al. (2018), so-called centralised critic and decentralised execution (CT-DE) methods use a centralised critic whose role is to estimate the agents' expected returns. The critic * Correspondence to

show abstract

“…Our setup is related to stochastic differential games with impulse control[23,9]. However, our Markov Game (MG) differs markedly since it is nonzero-sum, an agent assumes control and is a discrete-time treatment.…”

mentioning

confidence: 99%

SEREN: Knowing When to Explore and When to Exploit

Mguni¹,

Liu²,

Sootla³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Efficient reinforcement learning (RL) involves a trade-off between "exploitative" actions that maximise expected reward and "explorative'" ones that sample unvisited states. To encourage exploration, recent approaches proposed adding stochasticity to actions, separating exploration and exploitation phases, or equating reduction in uncertainty with reward. However, these techniques do not necessarily offer entirely systematic approaches making this trade-off. Here we introduce SElective Reinforcement Exploration Network (SEREN) that poses the exploration-exploitation trade-off as a game between an RL agent-Exploiter, which purely exploits known rewards, and another RL agent-Switcher, which chooses at which states to activate a pure exploration policy that is trained to minimise system uncertainty and override Exploiter. Using a form of policies known as impulse control, Switcher is able to determine the best set of states to switch to the exploration policy while Exploiter is free to execute its actions everywhere else. We prove that SEREN converges quickly and induces a natural schedule towards pure exploitation. Through extensive empirical studies in both discrete (MiniGrid) and continuous (MuJoCo) control benchmarks, we show that SEREN can be readily combined with existing RL algorithms to yield significant improvement in performance relative to state-of-the-art algorithms.

show abstract

A Viscosity Approach to Stochastic Differential Games of Control and Stopping Involving Impulsive Control

Cited by 6 publications

References 24 publications

Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints

Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints

LIGS: Learnable Intrinsic-Reward Generation Selection for Multi-Agent Learning

SEREN: Knowing When to Explore and When to Exploit

Contact Info

Product

Resources

About