An Adaptive Strategy Selection Method With Reinforcement Learning for Robotic Soccer Games

Shi, Haobin; Lin, Zhiqiang; Hwang, Kao-Shing; Yang, Shike; Chen, Jialin

doi:10.1109/access.2018.2808266

Cited by 36 publications

(17 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For Markov dynamic systems, the stochastic control problems are modeled as Semi-Markov decision processes (SMDPs) [26,27]. The time cost for the RL system to transit from one state to the next state is defined as the sojourn time.…”

Section: An Improved Q-learning Methods In Semi-markov Decision Processesmentioning

confidence: 99%

Fuzzy Reinforcement Learning and Curriculum Transfer Learning for Micromanagement in Multi-Robot Confrontation

Meng

2019

Information

View full text Add to dashboard Cite

Multi-Robot Confrontation on physics-based simulators is a complex and time-consuming task, but simulators are required to evaluate the performance of the advanced algorithms. Recently, a few advanced algorithms have been able to produce considerably complex levels in the context of the robot confrontation system when the agents are facing multiple opponents. Meanwhile, the current confrontation decision-making system suffers from difficulties in optimization and generalization. In this paper, a fuzzy reinforcement learning (RL) and the curriculum transfer learning are applied to the micromanagement for robot confrontation system. Firstly, an improved Qlearning in the semi-Markov decision-making process is designed to train the agent and an efficient RL model is defined to avoid the curse of dimensionality. Secondly, a multi-agent RL algorithm with parameter sharing is proposed to train the agents. We use a neural network with adaptive momentum acceleration as a function approximator to estimate the state-action function. Then, a method of fuzzy logic is used to regulate the learning rate of RL. Thirdly, a curriculum transfer learning method is used to extend the RL model to more difficult scenarios, which ensures the generalization of the decision-making system. The experimental results show that the proposed method is effective.

show abstract

Section: An Improved Q-learning Methods In Semi-markov Decision Processesmentioning

confidence: 99%

Fuzzy Reinforcement Learning and Curriculum Transfer Learning for Micromanagement in Multi-Robot Confrontation

Meng

2019

Information

View full text Add to dashboard Cite

show abstract

“…A complete solution for decision making in a robotic soccer environment consists in aggregating the training dataset instances and the selection of adequate strategies using information from the environment, even if they are incomplete or imprecise [27]. This work combines Support Vector Machine (SVM), decision tree and reinforcement learning to generalize a complete control policy to the MRS.…”

Section: Related Workmentioning

confidence: 99%

“…Many works have developed solutions to create setplays in the robotic soccer domain [18][10] [1] [11][4] [27]. Section 2 presents details about these solutions and tools.…”

Section: Introductionmentioning

confidence: 99%

A Dataset Schema for Cooperative Learning from Demonstration in Multi-robot Systems

Simões

Silva

Nogueira

2019

J Intell Robot Syst

View full text Add to dashboard Cite

Multi-Agent Systems (MASs) have been used to solve complex problems which demand intelligent agents working together to reach the desired goals. These Agents should effectively synchronize their individual behaviors so that they can act as a team in a coordinated manner to achieve the common goal of the whole system. One of the main issues in MASs is the agents' coordination, being common domain experts observing MASs execution disapprove agents' decisions. Even if the MAS was designed using the best methods and tools for agents' coordination, this difference of decisions between experts and MAS is confirmed. Therefore, this paper proposes a new dataset schema to support learning the coordinated behavior in MASs from demonstration. The results of the proposed solution are validated in a Multi-Robot System (MRS) organizing a collection of new cooperative plans recommendations from the demonstration by domain experts. Keywords Multi-Agent System · Learning from Demonstration · Dataset · Coordination · Multi-robot plan · clustering PACS 07.05.Mh Dataset Schema for Cooperative Learning in a MRS 3Learning from Demonstration (LfD) has been used to learn robots basic skills or even very simple setplays in a MRS [11]. Although, there is no register of using LfD to learn complex setplays. Therefore, this work proposes using LfD to offer domain experts a chance to watch robotic soccer matches and suggest new setplays for each situation for which they think the MRS has made a bad decision.Section 3 presents the state-of-the-art for learning coordinated plans in MAS. One of the main issues in LfD for setplays is the nature of the dataset generated from the domain experts recommendation. Some features in this dataset are not of primitive types as scalars or strings, but some complex types, such as objects, structures, trees, etc. Thus, we also define a strategy presented in Section 2.4, to handle this kind of complex data.The proposed solution has a two-level dataset, detailed in Section 4. To assess the feasibility of using this dataset to support setplays learning, the Fuzzy C-Means (FCM) algorithm is used to organize setplays recommendations into clusters. The choice of the FCM algorithm is due to the imprecision inherent in the friendly interface proposed for use by experts to generate the recommendations of setplays. The suggestions from experts are organized in clusters to solve the semantic equivalence issue presented in Section 2. Section 5 describes the assessment process and its results. Section 6 has conclusion and future work descriptions.

show abstract

“…Robotic Soccer Games are popularly used in evaluating the reinforcement learning methods, such as [31]. There are many subproblems of the soccer games, such as the Keepaway [32].…”

Section: Keepaway Taskmentioning

confidence: 99%

Reusing Source Task Knowledge via Transfer Approximator in Reinforcement Transfer Learning

2018

View full text Add to dashboard Cite

Transfer Learning (TL) has received a great deal of attention because of its ability to speed up Reinforcement Learning (RL) by reusing learned knowledge from other tasks. This paper proposes a new transfer learning framework, referred to as Transfer Learning via Artificial Neural Network Approximator (TL-ANNA). It builds an Artificial Neural Network (ANN) transfer approximator to transfer the related knowledge from the source task into the target task and reuses the transferred knowledge with a Probabilistic Policy Reuse (PPR) scheme. Specifically, the transfer approximator maps the state of the target task symmetrically to states of the source task with a certain mapping rule, and activates the related knowledge (components of the action-value function) of the source task as the input of the ANNs; it then predicts the quality of the actions in the target task with the ANNs. The target learner uses the PPR scheme to bias the RL with the suggested action from the transfer approximator. In this way, the transfer approximator builds a symmetric knowledge path between the target task and the source task. In addition, two mapping rules for the transfer approximator are designed, namely, Full Mapping Rule and Group Mapping Rule. Experiments performed on the RoboCup soccer Keepaway task verified that the proposed transfer learning methods outperform two other transfer learning methods in both jumpstart and time to threshold metrics and are more robust to the quality of source knowledge. In addition, the TL-ANNA with the group mapping rule exhibits slightly worse performance than the one with the full mapping rule, but with less computation and space cost when appropriate grouping method is used.

show abstract

An Adaptive Strategy Selection Method With Reinforcement Learning for Robotic Soccer Games

Cited by 36 publications

References 35 publications

Fuzzy Reinforcement Learning and Curriculum Transfer Learning for Micromanagement in Multi-Robot Confrontation

Fuzzy Reinforcement Learning and Curriculum Transfer Learning for Micromanagement in Multi-Robot Confrontation

A Dataset Schema for Cooperative Learning from Demonstration in Multi-robot Systems

Reusing Source Task Knowledge via Transfer Approximator in Reinforcement Transfer Learning

Contact Info

Product

Resources

About