Competitive Multi-agent Deep Reinforcement Learning with Counterfactual Thinking

Abstract

Counterfactual thinking describes a psychological phenomenon that people re-infer the possible results with different solutions about things that have already happened. It helps people to gain more experience from mistakes and thus to perform better in similar future tasks. This paper investigates the counterfactual thinking for agents to find optimal decision-making strategies in multi-agent reinforcement learning environments. In particular, we propose a multi-agent deep reinforcement learning model with a structure which mimics the human-psychological counterfactual thinking process to improve the competitive abilities for agents. To this end, our model generates several possible actions (intent actions) with a parallel policy structure and estimates the rewards and regrets for these intent actions based on its current understanding of the environment. Our model incorporates a scenario-based framework to link the estimated regrets with its inner policies. During the iterations, our model updates the parallel policies and the corresponding scenario-based regrets for agents simultaneously. To verify the effectiveness of our proposed model, we conduct extensive experiments on two different environments with real-world applications. Experimental results show that counterfactual thinking can actually benefit the agents to obtain more accumulative rewards from the environments with fair information by comparing to their opponents while keeping high performing efficiency. Environment Agent a s r a=μ(s) Customers Sense and avoid collisionCompete for more market share RL process

‘What if I Would Have Done Otherwise…’: A Controlled Adaptive Network Model for Mental Models in Counterfactual Thinking

“…Counterfactual thinking has already been used as a source of inspiration to refine and optimize well-known Q-learning approaches to reinforcement learning to let agents in a multi-agent setting improve their competitive abilities. A study presented in (Wang et al 2019), showed that counterfactual thinking can make the agents obtain more accumulative rewards from the environments with fair information in comparison to their opponents.…”

‘If Only I Would Have Done that…’: A Controlled Adaptive Network Model for Learning by Counterfactual Thinking

“…Counterfactual thinking has already been used as a source of inspiration to refine and optimize well-known Q-learning approaches to reinforcement learning to let agents in a multi-agent setting improve their competitive abilities. A study done by Wang et al [23], showed that counterfactual thinking can make the agents obtain more accumulative rewards from the environments with fair information in comparison to their opponents.…”

Cross-View Geo-Localization via Learning Disentangled Geometric Layout Correspondence

“…The idea of counterfactual in causal inference [32] has been successfully applied in several research areas such as explainable artificial intelligence [33], visual question answering [34], physics simulation [35], and reinforcement learning [36]. Inspired by these recent successes in counterfactual causal inference, in our preliminary paper [1], we propose a novel distance-based counterfactual (CF) learning schema that strengthens the quality of learned geometric descriptors for our GeoDTR+ and keeps it away from an obvious 'wrong' solution.…”