On-policy concurrent reinforcement learning

Abstract

When an agent learns in a multi-agent environment, the payoff it receives is dependent on the behaviour of the other agents. If the other agents are also learning, its reward distribution becomes non-stationary. This makes learning in multi-agent systems more difficult than single-agent learning. Prior attempts at value-function based learning in such domains have used off-policy Q-learning that do not scale well as the cornerstone, with restricted success. This paper studies on-policy modifications of such algorithms, with the promise of scalability and efficiency. In particular, it is proven that these hybrid techniques are guaranteed to converge to their desired fixed points under some restrictions. It is also shown, experimentally, that the new techniques can learn (from self-play) better policies than the previous algorithms (also in self-play) during some phases of the exploration.

Adaptive and self-configurable honeypots

“…According to Junling Hu et al the algorithmic complexity of finding an equilibrium in matrix games is unknown. Bikramjit et al prove that the minmax-Q and Nash-Q are equivalent in the purely competitive domains [13]. However, the advantage of the minmax-Q algorithm that it is more resource-efficient.…”

“…However, a problem of such an approach is that attackers are considered as part of the environment despite their competitive nature. According to Banerjee et al [13] considering competitors in a learning scenario as aspects of the environment may mean the environment is no longer stationary and convergence results may be impacted.…”

“…In such a situation, agents need to explore the environment and their objective is to find an optimal policy that maximizes their expected rewards. According, to Banerjee et al [13] the dis tributed reward may depend on the behaviors of the opponents, which makes reward computation challenging, because each opponent has his or her own self-interests. 1) Attacker reward: Assume that an attacker has a dedi cated goal, denoted s*, while penetrating the system.…”

“…The purpose of an adaptive high-interaction honeypot is to incrementally learn the optimal policy for choosing actions in given states. In the context of adaptive honeypots we use the minmax-learning proposed by Banerjee et al [13] and define a stochastic game between an attacker and a honeypot. As we have two agents, lets denote a player k and k the competitor.…”

Adaptive policy gradient in multiagent learning

“…This makes the focus on multiagent learning research extremely timely and justified. Several algorithms for multiagent learning have been proposed [2,5,7,8], mostly guaranteed to converge to an equilibrium in the limit. It is noted in [3] that none of these methods simultaneously satisfies rationality and convergence, two of the desirable criteria for any multiagent learning algorithm.…”