Reinforcement learning for autonomic network repair

Abstract

We report on our efforts to formulate autonomic network repair as a reinforcement-learning problem. Our implemented system is able to learn to efficiently restore network connectivity after a failure.Our research explores a reinforcement-learning (Sutton & Barto 1998) formulation we call cost-sensitive fault remediation (CSFR), which was motivated by problems that arise in sequential decision making for diagnosis and repair. We have considered problems of web-server maintenance and disk-system replacement, and have fully implemented an experimental network-repair application.In cost-sensitive fault remediation, a decision maker is responsible for repairing a system when it breaks down. To narrow down the source of the fault, the decision maker can perform a test action at some cost, and to repair the fault it can carry out a repair action. A repair action incurs a cost and either restores the system to proper functioning or fails. In either case, the system informs the decision maker of the outcome. The decision maker seeks a minimum cost policy for restoring the system to proper functioning.We can find an optimal repair policy via dynamic programming. Let B be the power set of the set of fault states S, which is the set of belief states of the system. For each b ∈ B, define the expected value of action a in belief state s as the expected cost of the action plus the value of the resulting belief state:Here, b i is the belief state resulting from taking action a in belief state b and obtaining outcome i ∈ {0, 1}; it is the subset of b consistent with this outcome. If a is a repair action and i = 1, we define the future value V (b 1 ) = 0, as there is no additional cost incurred once a repair action is successful. In all other cases, the value of a belief state is the minimum action value taken over all available choices:The quantities Pr(b) and c(b, a) are the prior probability of a belief state and expected cost of an action, which can be computed easily from a CSFR specification of the problem. Table 1 illustrates a small CSFR example with two fault states, A and B. The planning process for this example begins with the belief state {A, B}. It considers the test actions DefaultGateway and PingIP and the repair actions FixIP, UseCachedIP, and RenewLease. It does not consider DnsLookup since the action neither provides information (always 0), nor has a non-zero chance of repair. In evaluating the action PingIP, the algorithm finds that outcome 0 has a probability of .25 and outcome 1 has a probability of .75. Its expected cost from belief state {A, B} is then .25(50 + cost({A})) + .75(250 + cost({B})). (1) The expected cost from belief state {A} is computed recursively. Since all test actions have an outcome with an estimated probability of 0, only repair actions are considered. Of these, RenewLease is chosen as the optimal action, with

Scheduling Design and Verification for Open Soft Real-Time Systems

“…A related technique, Reinforcement Learning (RL) [15] (often called Approximate Dynamic Programming), has been applied to several domains, including computer cluster management [16] and network configuration repair [17], and job scheduling [18]. An eventual goal of our work is to support scheduler design using techniques like RL for open soft real-time systems, but that topic is beyond the scope of this paper.…”

Detecting Triangle Inequality Violations in Internet Coordinate Systems by Supervised Learning

“…Unlike previous heuristics-based work, we look for discriminative variables by using supervised learning methods, the use of which has become a trend in the field of networking [12,13]. In particular, we collect training data from popular ICS algorithms such as Vivaldi [5] and extract as many variables of different kinds as possible.…”

Scheduling for Reliable Execution in Autonomic Systems

“…A related technique, Reinforcement Learning (RL) [22] (often called Approximate Dynamic Programming), has been identified as a learning technology that holds great promise for the autonomic computing community [23]. It has been successfully been applied to several domains, including computer cluster management [24] and network configuration repair [25], and job scheduling [26]. However, RL algorithms are typically iterative and, in practice provide an approximation to the optimal solution.…”