A Data Set for Fault Detection Research on Component-Based Robotic Systems

Abstract

Abstract. Fault detection and identification methods (FDI) are an important aspect for ensuring consistent behavior of technical systems. In robotics FDI promises to improve the autonomy and robustness. Existing FDI research in robotics mostly focused on faults in specific areas, like sensor faults. While there is FDI research also on the overarching software system, common data sets to benchmark such solutions do not exist. In this paper we present a data set for FDI research on robot software systems to bridge this gap. We have recorded an HRI scenario with our RoboCup@Home platform and induced diverse empirically grounded faults using a novel, structured method. The recordings include the complete event-based communication of the system as well as detailed performance counters for all system components and exact ground-truth information on the induced faults. The resulting data set is a challenging benchmark for FDI research in robotics which is publicly available.

Performance regression testing and run-time verification of components in robotics systems

“…Depending on the affected subsystems and the application domain, the outcomes of these effects can range from the unnecessary consumption of energy or vanishing user acceptance to severe injuries in case of safety-critical systems. Surveys presented by Steinbauer [3] and in our own work [4] have shown the existence and relevance of such issues for robotics systems. Therefore, the computational performance of robotics systems needs to be closely observed and suitable tools are required to detect performance regressions as soon as possible during development.…”

“…This way, recording results can be replayed completely for detailed analysis and debugging purposes. The recording method has previously been described in Wienke et al [4], including details about the performance counters. Figure 5 shows an excerpt of a test case recording for a single test case with two test phases which are executed for different parameters.…”

Autonomous fault detection for performance bugs in component-based robotic systems

“…EVALUATION In order to validate our approach we have evaluated it on a previously published data set [19] for evaluating fault detection approaches in component-based robotic systems. Up to our knowledge, no suitable data set with the targeted fault categories, event data, performance counters and accurate ground truth existed before, which made it necessary to recored a new one.…”

“…It consists of 10 fault-free runs of the system and 33 runs with induces faults. For further details, please refer to Wienke et al [19].…”

“…ROC curves as micro-averages across components. For details about the individual faults please refer to Wienke et al [19].…”