Improving Evaluation Methodology for Autonomous Surface Vessel COLREGS Compliance

Stankiewicz, Paul G.; Mullins, Galen E.

doi:10.1109/oceanse.2019.8867549

Cited by 15 publications

(8 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…13 There also exist previous works in automatic evaluation of maritime Collision Avoidance (CA) systems for autonomous ships, including Woerner, 14 which uses tailored penalty functions. This approach is further developed by Pedersen et al 3 Stankiewicz and Mullins 15 propose a different approach by running a large number of simulations and mapping decision boundaries using clustering methods. Lee et al 16 present a falsification method called Adaptive Stress Testing for aircraft collision avoidance, which uses an adversary reinforcement learning based agent for the environment to identify falsifying interactions.…”

Section: Introductionmentioning

confidence: 99%

Automatic simulation-based testing of autonomous ships using Gaussian processes and temporal logic

Torben

Glomsrud

Pedersen

et al. 2022

Proceedings of the Institution of Mechanical Engineers, Part O:

View full text Add to dashboard Cite

A methodology for automatic simulation-based testing of control systems for autonomous vessels is proposed. The work is motivated by the need for increased test coverage and formalism in the verification efforts. It aims to achieve this by formulating requirements in the formal logic Signal Temporal Logic (STL). This enables automatic evaluation of simulations against requirements using the STL robustness metric, resulting in a robustness score for requirements satisfaction. Furthermore, the proposed method uses a Gaussian Process (GP) model for estimating robustness scores including levels of uncertainty for untested cases. The GP model is updated by running simulations and observing the resulting robustness, and its estimates are used to automatically guide the test case selection toward cases with low robustness or high uncertainty. The main scientific contribution is the development of an automatic testing method which incrementally runs new simulations until the entire parameter space of the case is covered to the desired confidence level, or until a case which falsifies the requirement is identified. The methodology is demonstrated through a case study, where the test object is a Collision Avoidance (CA) system for a small high-speed vessel. STL requirements for safety distance, mission compliance, and COLREG compliance are developed. The proposed method shows promise, by both achieving verification in feasible time and identifying falsifying behaviors which would be difficult to detect manually or using brute-force methods. An additional contribution of this work is a formalization of COLREG using temporal logic, which appears to be an interesting direction for future work.

show abstract

Section: Introductionmentioning

confidence: 99%

Automatic simulation-based testing of autonomous ships using Gaussian processes and temporal logic

Torben

Glomsrud

Pedersen

et al. 2022

Proceedings of the Institution of Mechanical Engineers, Part O:

View full text Add to dashboard Cite

show abstract

“…Most relevant to LEAS, Mullins et al [27] provide a tool which automatically identifies test cases for a system under test with a search based optimization approach dependent on a set of mission scenario configurations and a performance score for each configuration. A case study using the aforementioned tool in an autonomous surface vessel domain can be found in [77]. Bridging the gap between formal verification and automated test case generation, Akellea et al [78] provide a black-box method to identify test cases which do not satisfy a provided temporal logic specification based on a dataset of observed demonstrations.…”

Section: Automated Test Generationmentioning

confidence: 99%

A Mapping of Assurance Techniques for Learning Enabled Autonomous Systems to the Systems Engineering Lifecycle

Ellis

Wigness

Fiondella

2022

2022 IEEE International Conference on Assured Autonomy (ICAA)

View full text Add to dashboard Cite

Learning enabled autonomous systems provide increased capabilities compared to traditional systems. However, the complexity of and probabilistic nature in the underlying methods enabling such capabilities present challenges for current systems engineering processes for assurance, and test, evaluation, verification, and validation (TEVV). This paper provides a preliminary attempt to map recently developed technical approaches in the assurance and TEVV of learning enabled autonomous systems (LEAS) literature to a traditional systems engineering v-model. This mapping categorizes such techniques into three main approaches: development, acquisition, and sustainment. We review the latest techniques to develop safe, reliable, and resilient learning enabled autonomous systems, without recommending radical and impractical changes to existing systems engineering processes. By performing this mapping, we seek to assist acquisition professionals by (i) informing comprehensive test and evaluation planning, and (ii) objectively communicating risk to leaders.

show abstract

“…The approach proposed in the current article follows a similar idea. One such approach by Stankiewicz and Mullins [11] evaluates the overall performance of the autonomous ship by combining different subscores and assessing mission accomplishment, safety and International Regulations for Preventing Collisions at Sea (COLREGs) compliance. These sub-scores are combined into the final score using a weighted sum method.…”

Section: Related Workmentioning

confidence: 99%

Evaluation Simulator Platform for Extended Collision Risk of Autonomous Surface Vehicles

Vagale

2022

JMSE

View full text Add to dashboard Cite

Autonomous surface vehicles need to be at least as safe as conventional vessels, if not safer, when navigating on waters. With a great deal of navigation algorithms for surface vessels out there, the safety of their produced paths is questionable, and, in most cases, complicated to assess and compare. Hence, this paper proposes a method for extended collision risk assessment for paths generated by autonomous navigation algorithms as follows: (1) static, dynamic, and historic risk factors are calculated; (2) individual risk value is determined using a fuzzy inference system; (3) the extended collision risk assessment (ECRA) score is acquired using a root-mean-square method. Finally, a comparison of the ECRA score of each path determines the path with the lowest risk. The validation results show that the proposed method is able to detect lower/higher risk scenarios and assign an adequate risk value in most cases. Risk reduction for cautious paths varies up to 8.43%, while risk increases for incautious paths—up to 57.98%. The results indicate that the method could be used for navigation algorithm evaluation and comparison with some improvements. This research also reveals several promising future directions and applications of the method.

show abstract

Improving Evaluation Methodology for Autonomous Surface Vessel COLREGS Compliance

Cited by 15 publications

References 14 publications

Automatic simulation-based testing of autonomous ships using Gaussian processes and temporal logic

Automatic simulation-based testing of autonomous ships using Gaussian processes and temporal logic

A Mapping of Assurance Techniques for Learning Enabled Autonomous Systems to the Systems Engineering Lifecycle

Evaluation Simulator Platform for Extended Collision Risk of Autonomous Surface Vehicles

Contact Info

Product

Resources

About