Verifiable and Scalable Mission-Plan Synthesis for Autonomous Agents

Abstract

The problem of synthesizing mission plans for multiple autonomous agents, including path planning and task scheduling, is often complex. Employing model checking alone to solve the problem might not be feasible, especially when the number of agents grows or requirements include real-time constraints. In this paper, we propose a novel approach called MCRL that integrates model checking and reinforcement learning to overcome this limitation. Our approach employs timed automata and timed computation tree logic to describe the autonomous agents' behavior and requirements, and trains the model by a reinforcement learning algorithm, namely Q-learning, to populate a table used to restrict the state space of the model. Our method provides means to synthesize mission plans for multi-agent systems whose complexity exceeds the scalability boundaries of exhaustive model checking, but also to analyze and verify synthesized mission plans to ensure given requirements. We evaluate the proposed method on various scenarios involving autonomous agents, as well as present comparisons with two similar approaches, TAMAA and UPPAAL STRATEGO. The evaluation shows that MCRL performs better for a number of agents greater than three.

Correctness-guaranteed strategy synthesis and compression for multi-agent autonomous systems

“…To overcome these challenges of MAS planning, we design an approach called MoCReL, which is an improved version of MCRL that we have proposed previously [12]. MCRL combines model checking with reinforcement learning, so it can deal with more agents than the algorithmic methods do, however, its task types do not support collaborations and events in MCRL, and large plans cannot be compressed either.…”

“…As a major difference between MoCReL and our previous approach MCRL [12], the models in MoCReL are much easier to adapt to different scenarios of the planning problem, as they are instantiated from model templates. One does not need to change the templates but only instantiate the templates with different values of parameters in order to fit in one's own application.…”

“…Different from MCRL [12], the verification in MoCReL is directly conducted on the MAS model under the control of a strategy, because Uppaal Stratego supports the following verification queries [15]: Algorithm 2: Algorithm of liveness verification (adapted from Fig. 3 in the literature [32]): model checking G | σ against Query (9).…”

“…3. strategy policy = minE(time-caught×REWARD)[<=100]{robot.location}-->{}:<> time>=90 (11) A<> caught under policy (12) A[] !collide() under policy (13) Query (11) synthesizes a strategy named policy, which allows the robot to catch the cat within 100 time units ([<=100]). The condition at the end of the query, i.e., time >= 90, specifies when to sample runs.…”

“…We have previously proposed MCRL (Model Checking + Reinforcement Learning) [12,13] as a method that combines model checking with reinforcement learning to synthesize and verify plans of agents. MCRL benefits from both model checking and reinforcement learning so that the scale of the problem that MCRL can solve is larger than that of search-based methods, and also the results (a.k.a., plans) are guaranteed to be correct by model checking [11,13,14].…”