Modeling the effects of environmental and perceptual uncertainty using deterministic reinforcement learning dynamics with partial observability

Barfuß, Wolfram; Mann, Richard P.

doi:10.1103/physreve.105.034409

Cited by 8 publications

(10 citation statements)

References 86 publications

(92 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thirdly, noise in the perceptions of agents can affect cooperation in learning and evolutionary dynamics 70 . For example, inaccurate observations can lead to better learning outcomes in faster learning time, the stabilization of an otherwise chaotic learning process, and the mitigation of social dilemmas 71 (Fig. SI 1 ).…”

Section: Discussionmentioning

confidence: 99%

Intrinsic fluctuations of reinforcement learning promote cooperation

Barfuß

Meylahn

2023

Sci Rep

Self Cite

View full text Add to dashboard Cite

In this work, we ask for and answer what makes classical temporal-difference reinforcement learning with $$\epsilon$$ ϵ -greedy strategies cooperative. Cooperating in social dilemma situations is vital for animals, humans, and machines. While evolutionary theory revealed a range of mechanisms promoting cooperation, the conditions under which agents learn to cooperate are contested. Here, we demonstrate which and how individual elements of the multi-agent learning setting lead to cooperation. We use the iterated Prisoner’s dilemma with one-period memory as a testbed. Each of the two learning agents learns a strategy that conditions the following action choices on both agents’ action choices of the last round. We find that next to a high caring for future rewards, a low exploration rate, and a small learning rate, it is primarily intrinsic stochastic fluctuations of the reinforcement learning process which double the final rate of cooperation to up to 80%. Thus, inherent noise is not a necessary evil of the iterative learning process. It is a critical asset for the learning of cooperation. However, we also point out the trade-off between a high likelihood of cooperative behavior and achieving this in a reasonable amount of time. Our findings are relevant for purposefully designing cooperative algorithms and regulating undesired collusive effects.

show abstract

Section: Discussionmentioning

confidence: 99%

Intrinsic fluctuations of reinforcement learning promote cooperation

Barfuß

Meylahn

2023

Sci Rep

Self Cite

View full text Add to dashboard Cite

show abstract

“…Therefore, it is important to explore how the perception of partially true information affects the evolution of cooperation. On the other hand, instead of blurring the perception of the level of cooperation [22], misinformation may affect the individual perception of the game type [41]. It would be practical to further study the effect of this type of misinformation on cooperation dynamics in structured populations.…”

Section: Discussionmentioning

confidence: 99%

“…Nevertheless, to analyse the impact of misinformation on collective behaviours, there are many different frameworks [22,41]. For instance, a previous study exploring the effect of perception bias on cooperation dynamics where individuals adapt their behaviour through the best response based on their estimates of the current distribution of strategies [22].…”

Section: Discussionmentioning

confidence: 99%

Impact of misinformation in the evolution of collective cooperation on networks

Meng,

Broom,

2023

J. R. Soc. Interface.

View full text Add to dashboard Cite

Human societies are organized and developed through collective cooperative behaviours. Based on the information in their environment, individuals can form collective cooperation by strategically changing unfavourable surroundings and imitating superior behaviours. However, facing the rampant proliferation and spreading of misinformation, we still lack systematic investigations into the impact of misinformation on the evolution of collective cooperation. Here, we study this problem by classical evolutionary game theory. We find that the existence of misinformation generally impedes the emergence of collective cooperation on networks, although the level of cooperation is slightly higher for weak social cooperative dilemma below a proven threshold. We further show that this possible advantage diminishes as social connections become denser, suggesting that the detrimental effect of misinformation further increases when ‘social viscosity’ is low. Our results uncover the quantitative effect of misinformation on suppressing collective cooperation, and pave the way for designing possible mechanisms to improve collective cooperation.

show abstract

“…To investigate the learning dynamics for a learning rate of α < 1, we refine the mathematical technique of deterministic reinforcement learning dynamics (40)(41)(42)(43). This method considers the mean-field of an infinite memory batch to construct idealized learning updates precisely in the direction of the strategy-average temporal-difference error.…”

Section: Dissecting Reinforcement Learningmentioning

confidence: 99%