Dynamic Resource Allocation in Systems-of-Systems Using a Heuristic-Based Interpretable Deep Reinforcement Learning

Abstract

Systems of systems often include multiple agents that interact in both cooperative and competitive modes. Moreover, they involve multiple resources, including energy, information, and bandwidth. If these resources are limited, agents need to decide how to share resources cooperatively to reach the system-level goal, while performing the tasks assigned to them autonomously. This paper takes a step towards addressing these challenges by proposing a dynamic two-tier learning framework, based on Deep Reinforcement Learning that enables dynamic resource allocation while acknowledging the autonomy of systems constituents. The two-tier learning framework that decouples the learning process of the SoS constituents from that of the resource manager ensures that the autonomy and learning of the SoS constituents are not compromised as a result of interventions executed by the resource manager. We apply the proposed two-tier learning framework on a customized OpenAI Gym environment and compare the results of the proposed framework to baseline methods of resource allocation to show the superior performance of the two-tier learning scheme across a different set of SoS key parameters. We then use the results of this experiment and apply our heuristic inference method to interpret the decisions of the resource manager for a range of environment and agent parameters.

The SoS conductor: Orchestrating resources with iterative agent‐based reinforcement learning

“…This process continues for a predetermined number of cycles or until convergence is achieved. This approach not only enhances the basic flat autonomy model from our previous work 5 but also introduces a new dimension of dynamism and flexibility. It is also particularly adept at managing multiple layers of autonomy and accommodating diverse SoS architectures.…”

“…Moreover, most of these studies primarily focus on the framework's general performance while scarcely exploring the emergence of coordination behavior, another vital aspect of multi-agent interactions in SoS. As our previous work 5 is mainly centered on the performance and explainability of the proposed Two-Tier framework, this paper complements the additional aspect of incorporating coordination functionality in SoS within our Two-Tier framework. This approach can facilitate not only system constituents but also resource managers at the subsystem level to coordinate effectively with each other.…”

“…As a result, CTDE further destabilizes training, leading to poor performance in practice. Therefore, while acknowledging the merits and advantages of CTDE, our previous work 5 chooses DTDE as our learning method.…”

“…IAB-RL training occurs only atTier II, where managers learn to coordinate with others for limited resources and efficiently allocate their available resources. For Tier I, the learning process is similar to ref [5],. where system constituents in different sub-systems identify their target landmarks and learn how to spread faster without colliding in the modified OpenAI gym particle environment.The main idea of IAB-RL is to allow only one agent to learn and fix the policies of other agents at each iteration, avoiding the nonstationary problem in DTDE MARL.…”

“…It is important to emphasize that the learned policies are tested in a zeroshot manner59 (without further fine-tuning for a new task), as we train resource managers with 7.5 units of resource budget and test them on tasks with different resource budgets without additional fine-tuning.As shown in the figure, we use a random resource allocation strategy as the baseline, where the resource manager allocates different types of resources randomly until the available budget is reached, after which no resources are used. As discussed in our previous work,5 even an SoS with a single manager is too complex to utilize other methods or generate effective heuristics for solving this resource allocation problem, and our current framework comprises multiple sub-systems, making it more challenging. Most other RL methods related to resource allocation in SoS use more static and compact models of agents' behavior, rendering them suitable for narrower contexts and difficult to generalize as a baseline for evaluating our framework.…”