Time-Aware Test Case Execution Scheduling for Cyber-Physical Systems

Abstract

Testing cyber-physical systems involves the execution of test cases on target-machines equipped with the latest release of a software control system. When testing industrial robots, it is common that the target machines need to share some common resources, e.g., costly hardware devices, and so there is a need to schedule test case execution on the target machines, accounting for these shared resources. With a large number of such tests executed on a regular basis, this scheduling becomes difficult to manage manually. In fact, with manual test execution planning and scheduling, some robots may remain unoccupied for long periods of time and some test cases may not be executed. This paper introduces TC-Sched, a time-aware method for automated test case execution scheduling. TC-Sched uses Constraint Programming to schedule tests to run on multiple machines constrained by the tests' access to shared resources, such as measurement or networking devices. The CP model is written in SICStus Prolog and uses the Cumulatives global constraint. Given a set of test cases, a set of machines, and a set of shared resources, TC-Sched produces an execution schedule where each test is executed once with minimal time between when a source code change is committed and the test results are reported to the developer. Experiments reveal that TC-Sched can schedule 500 test cases over 100 machines in less than 4 minutes for 99.5% of the instances. In addition, TC-Sched largely outperforms simpler methods based on a greedy algorithm and is suitable for deployment on industrial robot testing.

Learning Objective Boundaries for Constraint Optimization Problems

“…Solving practical assignment, planning or scheduling problems often requires to repeatedly solve the same COP with different inputs [15,39,30,37]. Our approach is especially well-suited for those scenarios, where training data can be collected from previous iterations and historical data.…”

Rotational Diversity in Multi-Cycle Assignment Problems

“…As a second case study, we employ the real-world application of Test Case Selection and Assignment (TCSA) for cyberphysical systems (Yoo and Harman 2012), such as industrial robots. TCSA usually occurs in Continuous Integration (CI) processes, where new releases of the robot control software are regularly integrated and released (Mossige et al 2017 We consider four scenarios, 20 agents with 750, 1500, and 3000 test cases, and 30 agents with 3000 test cases. Table 2 summarizes the results.…”

“…As a second case study, we employ the real-world application of Test Case Selection and Assignment (TCSA) for cyberphysical systems (Yoo and Harman 2012), such as industrial robots. TCSA usually occurs in Continuous Integration (CI) processes, where new releases of the robot control software are regularly integrated and released (Mossige et al 2017) Additionally, the availability of agents is influenced by maintenance, technical faults, or short-term usage in other projects, and the set of test cases changes due to the ongoing development. Therefore, TCSA cannot be solved by a static assignment without the need for frequent updates.…”

Framework for Flexible, Adaptive Support of Test Management by Means of Software Agents

“…Test case scheduling addresses the challenge to find a suitable test case for each test resource or vice versa. In [21], this problem is tackled employing constrained programming. Even though this approach provides promising results for static distribution, dynamic distribution, i.e.…”