Dynamics of BoltzmannQlearning in two-player two-action games

Abstract

We consider the dynamics of Q learning in two-player two-action games with a Boltzmann exploration mechanism. For any nonzero exploration rate the dynamics is dissipative, which guarantees that agent strategies converge to rest points that are generally different from the game's Nash equlibria (NEs). We provide a comprehensive characterization of the rest point structure for different games and examine the sensitivity of this structure with respect to the noise due to exploration. Our results indicate that for a class of games with multiple NEs the asymptotic behavior of learning dynamics can undergo drastic changes at critical exploration rates. Furthermore, we demonstrate that, for certain games with a single NE, it is possible to have additional rest points (not corresponding to any NE) that persist for a finite range of the exploration rates and disappear when the exploration rates of both players tend to zero.

Melioration Learning in Two-Person Games

“…While equilibria are reached in some instances of the prisoner’s dilemma or the coordination game [41–43], the behaviour fails to converge in others. The results depend on the reward structure of the situation [44] as well as the particular version of Q-learning [45]. …”

Combined learning for resource allocation in autonomous heterogeneous cellular networks

“…The tradeoff between exploration and exploitation is generally a critical issue [10]. The most common method is to use a Boltzmann distribution, that is, the probability of choosing action an,jn E An at time t is given by 1A policy Yn = (Yn (an, 1 ), ... , Yn (an, J n )) is said to be stationary if it's not changing with time.…”

Decision Making Process of Hexapods in a Model of Complex Terrains

“…the number of iteration cycles. Based on previous experiments and recommendations [2,20,28], the values of the parameter are E 0 =E max =0.9, E min =0 a β=0.9. RL parameters and their descriptions can be found in [2,20].…”

“…The strategy used to select the action during learning is based on Boltzmann´s exploration [2,20,28]:…”