Validation and verification issues in a timeline-based planning system

Abstract

To foster effective use of artificial intelligence planning and scheduling (P&S) systems in the real world, it is of great importance to both (a) broaden direct access to the technology for the end users and (b) significantly increase their trust in such technology. Automated P&S systems often bring solutions to the users that are neither ‘obvious’ nor immediately acceptable to them. This is because these tools directly reason on causal, temporal, and resource constraints; moreover, they employ resolution processes designed to optimize the solution with respect to non-trivial evaluation functions. Knowledge engineering environments aim at simplifying direct access to the technology for people other than the original system designers, while the integration of validation and verification (V&V) capabilities in such environments may potentially enhance the users’ trust in the technology. Somehow, V&V techniques may represent a complementary technology, with respect to P&S, that contributes to developing richer software environments to synthesize a new generation of robust problem-solving applications. The integration of V&V and P&S techniques in a knowledge engineering environment is the topic of this paper. In particular, it analyzes the use of state-of-the-art V&V technology to support knowledge engineering for a timeline-based planning system called MrSPOCK. The paper presents the application domain for which the automated solver has been developed, introduces the timeline-based planning ideas, and then describes the different possibilities to apply V&V to planning. Hence, it continues by describing the step of adding V&V functionalities around the specialized planner, MrSPOCK. New functionalities have been added to perform both model validation and plan verification. Lastly, a specific section describes the benefits as well as the performance of such functionalities

Mission Operations and Autonomy

“…Both the model and the property are provided as inputs to model checking tool as shown in Figure 6.26. The translation from the MrSPOCK planning model to a model checker formal model requires introduction of a well-defined set of state variables and clocks [88]. State variables range on domain states, while clocks are used to represent time progression.…”

Flexible Plan Verification: Feasibility Results

“…A problem for a wider diffusion of such technology stems in the limited community that has been studying with formal methodologies the properties of this planning approach. These authors have been investigating the interconnection between timeline-based planning and standard techniques for formal validation and verification (V&V) with the aim of building a rich environment for knowledge engineering [6] and exploring properties that concern temporal plans and their execution [7].…”

Análise de pós-design para aplicações de planejamento em IA.

“…stakeholders, experts, users) still emerge after plan generation and execution. For example, the identification of unsatisfactory solutions (valid plans but undesirable from the users' perspective) and unbalanced trade-offs among different quality metrics and criteria (JÓNSSON, 2009;ENGELHARDT;CHIEN, 2000;CESTA et al, 2008) indicates a lack of understanding of the requirements and preferences in the model. These hidden requirements raise the need for iterative re-modeling and tuning process.…”

“…In some applications (such as space exploration (CESTA et al, 2008)) there are a number of criteria to evaluate a plan and complex trade-offs to be balanced. As discussed in Chapter 2, the definition of quality metric must be made wisely and carefully in AI planning problems.…”

“…A true test of our tool should be in the form of a case study with a real problem (e.g., (CESTA et al, 2008;JÓNSSON, 2009;). However, such case studies are not reproducible and often rely for success on external factors: significant interest from the client, success of other parts of the mission (e.g., landing on Mars), and larger economic forces.…”