Scenario co-evolution for reinforcement learning on a grid world smart factory domain

Gabor, Thomas; Sedlmeier, Andreas; Kiermeier, Marie; Phan, Thomy; Henrich, Marcel; Pichlmair, Monika; Kempter, Bernhard; Klein, Cornel; Sauer, Horst; Ag, Reiner SchmidSiemens; Wieghardt, Jan

doi:10.1145/3321707.3321831

Cited by 4 publications

(16 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To illustrate the definitions of the previous section, we introduce an example system called Grid World Smart Factory, which has also been used and implemented in [24]. How-ever, we first introduce a formal definition of a system for this domain.…”

Section: Example Domainmentioning

confidence: 99%

“…The first parts of the machine learning pipeline operate on a domain distribution, i.e., they are not specialized on a single instance of a use case but are designed to find models and solutions general enough to work on a range of similar tasks. Even when we only target a single domain eventually, having a decent amount of diversity during training is crucial to the success of machine learning [6,24,43]. During deployment, we switch from the more general distribution of possible domains to a more concrete instantiation fed with all the information we have about the deployed system and the environment it is deployed in.…”

Section: Software Engineering For Machine Learningmentioning

confidence: 99%

“…In Sect. 8, we apply this new notion to our example domain, formally explaining the results of [24]. In Sect.…”

Section: Introductionmentioning

confidence: 99%

“…We suggest to develop a system and its test suite in a competitive setting where each sub-system tries to outwit the other. We call this approach scenario coevolution, which we introduce formally and build the bridge to a practical application where it has already shown benefit [24]. We hope that trust in such dynamics can help to build a new process for quality assurance, even for hardly predictable systems.…”

Section: Introductionmentioning

confidence: 99%

“…3 by extending a formal framework for the description of systems first introduced in [28] and augment it to also include the process of software and system development. We use said framework to first present a formal definition of an example domain used in [24] in Sect. 4.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

The scenario coevolution paradigm: adaptive quality assurance for adaptive systems

Gabor

Sedlmeier

Phan

et al. 2020

Int J Softw Tools Technol Transfer

Self Cite

View full text Add to dashboard Cite

Systems are becoming increasingly more adaptive, using techniques like machine learning to enhance their behavior on their own rather than only through human developers programming them. We analyze the impact the advent of these new techniques has on the discipline of rigorous software engineering, especially on the issue of quality assurance. To this end, we provide a general description of the processes related to machine learning and embed them into a formal framework for the analysis of adaptivity, recognizing that to test an adaptive system a new approach to adaptive testing is necessary. We introduce scenario coevolution as a design pattern describing how system and test can work as antagonists in the process of software evolution. While the general pattern applies to large-scale processes (including human developers further augmenting the system), we show all techniques on a smaller-scale example of an agent navigating a simple smart factory. We point out new aspects in software engineering for adaptive systems that may be tackled naturally using scenario coevolution. This work is a substantially extended take on Gabor et al. (International symposium on leveraging applications of formal methods, Springer, pp 137-154, 2018).

show abstract

Section: Example Domainmentioning

confidence: 99%

Section: Software Engineering For Machine Learningmentioning

confidence: 99%

“…In Sect. 8, we apply this new notion to our example domain, formally explaining the results of [24]. In Sect.…”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

The scenario coevolution paradigm: adaptive quality assurance for adaptive systems

Gabor

Sedlmeier

Phan

et al. 2020

Int J Softw Tools Technol Transfer

Self Cite

View full text Add to dashboard Cite

show abstract

Resilient Multi-Agent Reinforcement Learning with Adversarial Value Decomposition

Phan

Belzner²,

Gabor

et al. 2021

AAAI

Self Cite

View full text Add to dashboard Cite

We focus on resilience in cooperative multi-agent systems, where agents can change their behavior due to udpates or failures of hardware and software components. Current state-of-the-art approaches to cooperative multi-agent reinforcement learning (MARL) have either focused on idealized settings without any changes or on very specialized scenarios, where the number of changing agents is fixed, e.g., in extreme cases with only one productive agent. Therefore, we propose Resilient Adversarial value Decomposition with Antagonist-Ratios (RADAR). RADAR offers a value decomposition scheme to train competing teams of varying size for improved resilience against arbitrary agent changes. We evaluate RADAR in two cooperative multi-agent domains and show that RADAR achieves better worst case performance w.r.t. arbitrary agent changes than state-of-the-art MARL.

show abstract

Discriminative Reward Co-Training

Altmann,

Ritz,

Zorn

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

We propose discriminative reward co-training (DIRECT) as an extension to deep reinforcement learning algorithms. Building upon the concept of self-imitation learning (SIL), we introduce an imitation buffer to store beneficial trajectories generated by the policy, determined by their return. A discriminator network is trained concurrently to the policy to distinguish between trajectories generated by the current policy and beneficial trajectories generated by previous policies. The discriminator’s verdict is used to construct a reward signal for optimizing the policy. By interpolating prior experience, DIRECT is able to act as a reward surrogate, steering policy optimization towards more valuable regions of the reward landscape, thus, towards learning an optimal policy. In this article we formally introduce the additional components, their intended purpose and parameterization, and define a unified training procedure. To reveal insights into the mechanics of the proposed architecture, we provide evaluations of the introduced hyperparameters. Further benchmark evaluations in various discrete and continuous control environments provide evidence that DIRECT is especially beneficial in environments possessing sparse rewards, hard exploration tasks, and shifting circumstances. Our results show that DIRECT outperforms state-of-the-art algorithms in those challenging scenarios by providing a surrogate reward to the policy and directing the optimization towards valuable areas.

show abstract

Scenario co-evolution for reinforcement learning on a grid world smart factory domain

Cited by 4 publications

References 30 publications

The scenario coevolution paradigm: adaptive quality assurance for adaptive systems

The scenario coevolution paradigm: adaptive quality assurance for adaptive systems

Resilient Multi-Agent Reinforcement Learning with Adversarial Value Decomposition

Discriminative Reward Co-Training

Contact Info

Product

Resources

About